Things you should avoid making a review

After the launch of the new Nexus I was eagerly waiting for the first reviews to find their way online.. and here they are!

I read Engadget, TechCrunch and The Verge reviews for both the Nexus 4 and the Nexus 10. The reviews are mostly positive although none of the reviewers sees the devices as perfect. Anyway this post is not about opinions on the Nexuses themselves. You can find some general thoughts on the Google announcements in my previous post.

The thing I'd like to address is review method. Both Engadget and TechCrunch specify somewhere in the articles that the units they are using don't have definitive software, they lack for example the multi user switching and the lock screen widgets implementation. The Verge doesn't mention such features so I suppose they don't have them either. Leaving aside the fact that if I were Google I wouldn't give away devices with non final software, what reviews are we talking about?

In my opinion calling those reviews is misleading: those are clearly Pre-Reviews because the software is not final. I can understand that the news business is all played around the first to print (or first online) but then you should state your the facts accordingly. In this case I would expect in the first lines something like "the review is based on pre release software so something could not translate to the final product and we'll update this article accordingly".

Beyond that you should be really careful about all the aspects of the review that could relate to the not definitive software. So, while the The Verge reviewers are quite satisfied with the battery life and performance of the devices, TechCrunch are particularly Engadget are not. Engadget makes the point loud and clear and adds it to the shortlist of the "Cons" of the Nexuses. Isn't anyone aware that this is an aspect that could be largely dependent on software optimization?

On this I'd point out also that benchmarks, if they have meaning for the customer experience at all, are really a joke on non final software. In fact I appreciate The Verge that bases the opinions on performance and battery life on qualitative remarks and daily usage more than synthetic benchmarks. 

That said nothing forbids that the battery issues will still stand on the final units but we'll need to wait for after release reviews for that.