Who Judges the Judgers?

Who Judges the Judgers?

Judging and scoring liquor is an act of subjectivity. It is like giving a score to a work of literature by a serious author; an acting performance in a movie; a meal at a fine restaurant; or a strutting starlet in a beauty pageant. Who is to say which is truly better? Who is to say why one person's tastes are superior or more authoritative than another?

Or, let's put this another way: no matter how many English professors call it the greatest novel ever written, few slog through the 1,000+ pages of Leo Tolstoy's War and Peace (and of those, fewer thank the critics when they're done). At the same time, only a vanishingly small number of "experts" will sing the praises of Anne Rice's Interview with a Vampire despite bestselling status. One can recycle that argument for Herman Melville vs Michael Crichton; drama vs comedy films in the academy awards; and country music vs Beethoven. Whatever blows your hair back, baby.

Yet here we are: a website absolutely dedicated to bringing these subjective scores on personal tastes to cyberspace. Most people appreciate it and take the scores for what they are... but have we heard some hate directed at the whole idea of liquor scoring? Yes. Yes we have indeed. These defenders of subjectivity are forceful and anything but shy. We've noticed four general themes:

  • The Burned Palate Complaint: "How can these people taste 20 or a 100 different things in one day and still give that 99th bottle a fair assessment? Your palate's burned to a crisp after that!" A fair argument perhaps... a critic may claim to be able to taste notes of shoe leather, dew-swept mountain grass, and newly turned earth on the lee side of a Belgian meadow... but to say all that after tasting 20 different vodkas before it? That judge is going to have to tell everyone that those are spirit tasting notes and not spiritual utterances born from his imagination.
  • The 'He Hate Me' Complaint (with apologies to ex-XFL'er Rod Smart): "There's no way these competitions are blind... they know exactly what they're tasting every single time. The small guys have no chance." Liquor is a multi-million dollar industry and it's hard to imagine that advertising, kickbacks, and other shenanigans don't play a role. The age (and price) of a spirit is often visible in the color and scent. Is a judge - no matter how "blind" - ready to give a $1,500 bottle of limited edition 27yr aged whiskey finished in wood from some endangered South Asian tree and blessed by the Italian Diocese on Easter Sunday a poorer score than a garden variety $15 bottle of scotch? Hard to say.
  • The Stinky Cheese Complaint: "It's the stinky cheese! You know, in a roomful of cheese you've got to stand out! So just like the stinky cheese always wins, the weirdest liquor always gets the best score. Then you drink it and you're like, 'What the heck is that all about?'" This is really a kind of extension of the Burned Palate Complaint in that subtle spirits tend to get drowned out in the noise and cacophony of large-scale tastings. The fairest and most well-meaning judge can't help but be swayed by the sight of a black swan in a flock of white.
  • The Dead Poet Society Complaint: "A number can't mean anything at all! What day of the week was it when he tasted it? What did he have for breakfast that morning? Are his kids flunking algebra? The scores change with the mood, the phases of the moon, whatever... and it's all complete BS." We've personally been assaulted on this issue, where someone deems scoring a subjective experience an act of madness akin to Robin Williams in the famous movie saying, "I like Byron, I give him a 42 but I can't dance to it," when ridiculing the idea of numerically scoring poetry.

Those are some pretty impressive arguments. Is it all really just a sham?

The cavalry rode to the defense of competitions when we ran into one of the regular judges from the San Francisco World Spirits Competition: David Mahoney, who is a wine and spirits writer for Minnesota Monthly. We ran into him at a tasting event in Chicago and, in an amazing portrayal of patience and good will, answered all of our rather pointed questions about the authenticity of the judging in San Francisco.

The judging is done in panels... last year there were 9 panels. "It starts out as a lot of fun," says David with a laugh. "It's a little bit like a party atmosphere: you know most of the other people and it's a great chance to taste some really great spirits. But by the end of the day... that can be tough." But he assures us that every judge is absolutely committed to giving every spirit as fair a tasting as humanly possible.

"And the most fun is when we really do get surprised by a low-priced gin or some other spirit. I remember drinking a really appealing scotch in a flight of similar ages and I loved it for its pear-like quality. It was regular old 12yr Glenffidich. I'd never have remembered how good it was if it hadn't been for that tasting. Same with Wembley Gin from this last year. It's really great!"

We wondered if being on a panel there's some arm-twisting that goes on. Can a majority of judges browbeat some holdouts into giving a higher score? "Sure, there's some influence if a highly respected judge is talking for or against something. But if there's any real disagreement, we set the spirit aside and come back to it later. We try very hard to give true, solid opinions and those double-golds really are unanimous opinions."

And is this test really blind? "Absolutely. 100% completely and totally blind. We have no idea when they're served." Come on, what about the highly distinctive stuff that's really unique? "Yeah, there are a few things that are categories unto themselves... there's no disguising that."

So look, what about saving your palate? After all, the competition tastes hundreds of spirits in a matter of days? "Water. We drink a ton of water. There's some bread too - pretty lousy, tasteless bread really. And some cheese. But mostly it's water. We spend a lot of time in the bathroom. Also, we depend a lot upon the nose of the spirit. You see, the nose doesn't fatigue like the palate does."

On the subject of scoring itself, David was very open and honest. "This is not a highly scientific process; we're still human beings making our best judgments at particular times of a given day... we're just a little more educated. But spirits today are a highly controlled product and everything made today is at a fairly high standard. Most of the things we try are very good. But we try and protect the gold medal... those spirits that I award a gold medal to I really feel are a class above the rest."

So which spirits are hardest to judge? "Bourbon. Bourbon is the hardest thing to judge. There's a familiarity with me and bourbon that makes it difficult."

And the easiest. "Vodka by far. I mean, come on, it's a writer's dream. You can say anything about vodka and not be wrong."

So of the complaints - the stinky cheeses and the burned palates and so forth - David Mahoney convinced us of a couple of things we thought we knew about judging in general and the San Francisco competition in particular. They really are blind; no question. The other issue is that the judges are fully aware of the pitfalls of subjectivity and time of day and burned out palates and so forth. And to the extent that any of these introduce errors into their judging, they go to great lengths to mitigate those problems. They take their scores - subjectivity and all - very, very seriously.

That brings us all the way back to what makes Proof66 special. When you can see a variety of scores from different times and different panels, that's when a true consensus begins to emerge separating good spirits from great ones (or highly unusual ones from more standard offerings). Even if there are subjective errors in given circumstances and situations, we'll take the considered opinion of many different David Mahoneys over time and accept that judgment as a guide to purchase each and every time.


2010-11-10
Published by Proof66.com