by Neal MacDonald, Editor
Business Insider published an article called Why You Should Only Buy Cheap Liquor written by Ben Taylor of FindtheBest. They came to the following conclusion in their article (our emphasis):
We compiled ratings and awards from the San Francisco World Spirits Competition, Wine Enthusiast Magazine, and the Beverage Testing Institute. We rated each bottle based on its performance at these competitions. Unlike grocery-store consumers, who tend to believe more expensive liquor tastes better, the experts at these competitions taste each bottle blind. They have no idea what the price of the liquor is, and can give a more unbiased rating as a result…
There's essentially zero correlation between price and quality.
They even made a nice chart graphing out scores to prove their point.
This is an entirely inappropriate conclusion. Here at Proof66, we have read every result from these judging institutions for the last 8 years. We have interviewed the judges. We have run our own statistical analyses on a variety of subjects. We have run the numbers to hone a very complex algorithm to address issues like consistency in scores and biases towards one particular style of spirit versus another. We have spent hours every single day for the last 5 years thinking about these numbers and what they mean. In short, we feel there is only a handful of people on this planet that know more about these scores from these judging institutions than we do.
So with that as context, here is why this conclusion is entirely inappropriate.
1) Their numbers don’t work. The “smart rating” that Business Insider is using is based upon the numbers from Find the Best… a review aggregation service. We don’t know anything about the algorithm that they run - we’re sure it’s proprietary - but at a glance we can determine that it doesn’t correct for issues that we find important: that is, consistency in scoring and spirit type. Why does this matter? At a glance, we can tell that they’re making no distinctions between when an award is given, from whom, or how the bell curve operates within the statistical set of judging results. They treat a double-gold medal result from San Francisco in 2013 is not worth the same as it was in 2012, or 2011, or any other year; but it’s not really the same. They treat a double-gold medal awarded in scotch as the same as one awarded for vodka or rum; but they’re not really equal. High scores from different institutions are given similar weight; but those judging criteria are not really equal. They also appear to be taking a general average of reviews, which means spirits are being punished for scoring high in one but low in another rather than sitting on their laurels. In short, these figures are treated far too lightly—almost like batting averages—in the sense of understanding how the subjective opinions of the judges come through in the numbers.
2) If their numbers did work, it still doesn’t work. Even if they used our algorithm at Proof66 they would find only a weak correlation between price and quality. But that’s still an inappropriate conclusion. If it was true, we’d have made similar proclamations ourselves; but we don’t. And here’s where own algorithm has to yield to some objective statistical and marketing issues and why we go to such pains to color it with user reviews:
In short, there’s nothing that makes sense about the conclusion from Business Insider when you look a little more closely at what’s going on.
So why do we do it here at Proof66?
Partly, because we believe in judging and critical assessment as a guide. We think that by raising the profile of these competitions they gain more market impact and encourage more participation. More participation will help supply a sample that truly represents the population… never perfectly, but better. We also think it happens to be better than the alternative, which is guesswork, advertising, and label appeal. Finally, we go through a lot of effort to try and balance out how those aggregated results appear so that they showcase the spirit in the most appealing way.
But most importantly, these scores are meant to guide an exploration of spirits. These results are designed primarily for the middle-market: those spirits in the $30 - $100 category that serve as the next rung up for people reaching out from big market brands. These scores are excellent at finding value in the middle market but a very poor assessment of the truly elite spirits. That’s the next level of connoisseurship when one steps beyond the middle-market bracket.
Business Insider did this because it’s easy. But it’s also very sloppy. A better conclusion would’ve been “How judging can help find great value in the middle-priced brands.” But that’s not at all as sexy a title as “Why You Should Only Buy Cheap Liquor.”
Here, Business Insider fell prey to the siren call of the sexy but untrue headline.