What I would like to see in a review

I've had this thought in my head for a while, and it's about reviews. Namely, for reviews of software packages, as doled out by the online (and print) community by the truckload. We love lists, and we love anything that can be used to quantify items, especially when it comes to our hard-earned dollars and where they go. The thing that bothers me is that, as a user community, we've lulled ourselves into accepting reviews that are as much "updates" to previous reviews as the software applications they hope to cover, and have lost an objective view (like not noticing we've put on a few pounds since 1996).

For example, Flash (formerly Macromedia, now under the Adobe umbrella... along with everything else) is probably the nuber one culprit on my list of reviews for which they mean less and less with each passing version upgrade. Every single time it gets reviewed, it gets glowing remarks, but always from the "Flash" expert on staff. Statements like "Oh, if you do a lot of Flash application development, you'll really love the enhancements made to this latest version." But, what if I don't do a lot of development in Flash? What if, in fact, I don't do ANYthing in Flash, but have heard so much, I would like to start. For the longtime Flash user, the latest version may be, in fact, 4 out of 5 stars or better, but we lose perspective on what it's like to start all over again. In fact, to the new user -- the one who's used tons of other programs in print design, but has never dabbled in Flash until now, when a prominent executive expects some Flash knowledge to permit a raise or a contract extension -- it's precipitous learning curve and more-code-than-animation functionality that has crept in over the years represented more of a 1 or 2 out of 5, at BEST.

It's certainly a difficult matter to resolve. Reviews are highly subjective to the reviewer, and when you're paying for a review, either through your subscription dollar or the attention you should be paying to a different income source (like a job), you expect a certain amount of reliable expertise. Except in a case such as this, I want a reviewer who knows as little about the application as I do, and warn me of what awaits me, with long nights spent combing through a "Learn Flash in 48 hours or Lose Your Job Trying."

Instead, here's an idea: try breaking reviews into two camps, the expert and the novice. Rather than spending an entire magazine page rehashing the product details found on the vendor's website, generate two reviews, one from the upgrader's point-of-view, and then one from the first-time user, and see how it pans out. I'd be willing to bet that many glaring issues and problems that have held over in some of the most long-standing apps, like Photoshop or Flash, would have their inefficiencies laid bare for the world to see when seen through the eyes of a new user. And let's be real here: I'm not talking a new user like "what is this mouse thingy" level. Just someone who doesn't use the app very often or at all, but may need to invest in it soon. I consider myself a power user in all of the Adobe apps (except maybe Acrobat... Acrobat is a one trick pony for me -- I get in, I get out), yet I want to throw my mouse at the screen whenever I have to do ANYthing in Flash. Why? Because, let's be honest, its interface is terrible. It's essentially the same frame-based animation tool from its days as Macromedia FutureSplash, and has had massive ActionScripting injections behind the scenes to reduce signs of aging. Wouldn't maybe a poor review from a fresh set of eyes perhaps spur Adobe to improve? Of course, the flip-side is that now, without any real competition, and an established ActionScripting base of developers, the urge to improve by the UI folks must be next to nil (something I suspect has happened over at Filemaker since version 2).

Anyway, my point still stands. I have always said new employees are a company's greatest resource, because they can spot all of the inefficiencies you and your staff have learned to live with day in and out. Encourage them to evaluate your company from the get-go, and weed out anything that seems hard to understand from their perspective, or learn where you can improve at least the transfer of knowledge form one generation to the next. If you plan to be a revered source by which people plan their investments, whether online or in print (or some other media which we have yet to see down the road), you need to offer expertise and innovation, or else run the risk of being averaged into the mix of all other reviews down the road.