Crowd-Sourced Ratings and Why They Suck

This isn’t going to be hard to write as some of what you read here, mainly because I’ve been dealing with this issue for ten solid years (almost daily) and really have no more thinking to do about it.

Crowd-sourced ratings – for anything; beer, whiskey, wine, chocolate, movies, restaurants, you name it – actually do the people they’re intended to help about as much good as they’d get from opening the yellow pages or a beverage catalog or standing in front of a store rack, closing their eyes, and picking at random. 

Before you ask, NO, this is not the rantings of a beverage critic who’s worried about his own credibility. The day I start even thinking of how my “cred” will be affected by what I write is the day I’ll shut this puppy down and go fishin’. I was first asked about the phenomenon of crowd-sourced reviews in 2003, by a wine sales rep who stumbled upon a site where European consumers reviewed wines. I remember visiting it (the thing is LONG gone by now) and finding two things: 1) the level of writing skill in Europe is absolutely no better than it is here in the US, and 2) the Euro crowd was every bit as prone to servicing their personal stylistic agendae as Americans. EVERY US wine rated on that site was accompanied by completely dismissive verbiage. “Drinkable” was the nicest remark I found. It was SO obviously slanted toward French and Italian wines (Spain got the same trashing as the US) that, unless you were squarely in that stylistic rut yourself, it was practically useless. The same names appeared next to dozens of reviews and it was, very quickly, possible to predict exactly what they’d rate a wine, just by the name and origins.

The lineal descendants of those fatheads populate the major rating sites of today: Yelp, RateBeer. BeerAdvocate, Epinions, and every other crowd-sourced review site. See for yourself: check how many 95 point+ reviews you find on the beer sites for any style of beer except IPA, Stout, Sour, Barleywine, Belgian Quads, Brett Ales, and one or two other stylistic niches. Check the scores for Pale Ales, Amber-Reds, ESB, Scottish or Scotch ales, Porters, Lagers in general, and Pilsners in particular. The readers and users of  these sites are almost always slaves to their personal preferences and current trends. It’s completely inarguable that nobody can reasonably rate a Pale Ale or an Amber by the same standard you’d apply to an Imperial IPA or a Flanders Red, just the same way as it’s impossible to legitimately compare the merits of a Napa Cabernet with a Washington Riesling or an Italian Sangiovese. Any of those are apples and oranges situations. And yet we’re led to believe that Pale Ales and Pilsners are lesser beers than the fawned-over Triple and Quad IPAs of the Moment. Pale Ales are their own thing and deserve to be judged versus the merits of their own style…and it’s just not done, in any general sense.

Todd Alstrom of BeerAdvocate

Group-think is a real thing; it’s been proven countless times in research and anecdotal studies. You get a group of six or seven (or fifteen) young men, 25 – 35, who visit breweries together on a regular basis, and there will be an Alpha Dog in the pack; maybe more than one. Those guys will set the stylistic preferences of the group and even those among them who may have not agreed with the Alphas to begin with will start to gravitate toward the common view, in order to be accepted. This also works among the forum responders on the rating websites. BeerAdvocate is notorious for its rowdy forum denizens, many of whom are firmly of the opinion that they know a LOT more about beer than the Alstrom brothers, the surprtisingly erudite siblings who own and run BA. (They’re wrong but you can’t tell ’em that.) I was ridiculed in the BeerAdvocate forums for – among other things – assigning a gaudy rating to Ninkasi “Imperiale”. One of the forumites responded with this: “You gave 99 points to that Ninkasi Imperiale crap! We already said what we think of that beer here!” The inference was clear: this person would never dream of liking a beer that was not properly anointed by his peer group, a form of lemming-like conformity which is emphatically NOT a part of my job.

Marvin Shanken, Wine Spectator publisher

I get offended messages from some of the Wine Spectator folks whenever I opine that I find their ratings fishy because of the fact that they run tasting groups. Quite aside from the indisputable fact that WS has become, as Dan Jenkins famously said of Sports Illustrated,  “a slick cookbook for your basic two–yacht family”, there is a fairly obvious uniformity to WS reviews, over the years, which seems impossible as a result of a collection of individual palates…unless they strongly influence one another. People in groups rarely tend to agree completely and yet we get Spectator reviews written by one of their number; reviews which stand as Spectator’s lasting opinion of that wine, raising the question of whose dissenting opinion did we not read. Highly erudite wine people tend to harbor certain prejudices: France is Supreme, certain styles of wine are inherently superior to others, and that inexplicable bell curve that says that wines with a sticker price of $40 and up are presumptively better than those below $20. In any group of people who evaluate anything, the goal is a majority opinion and, preferably, a uniformity of opinion. Squeaky wheels within such groups sometimes get the grease…and sometimes a tire change is the remedy.

Stephen Tanzer

I don’t mean to single out Wine Spectator, either. They’re just the most visible example and the one that touts their group tasting model most proudly. But in terms of how all this applies to you, the American wine consumer, I submit that it is far – as in light years far – more productive and reliable to find a like-minded individual critic, invest a bit of time to see if their palate is at all attuned to your own, and, if not, Move On. There is, as far as I know, no serious shortage of people out there who review beverages, from the uber-popular types like Anthony Dias Blue and Robert Parker and Steven Tanzer, to cranks like me and Doctor Vino. There is, literally, some common vibe to be found by everybody who’s trying to explore wine, from the gentlemanly work of my colleague, Paul Gregutt to the dyspeptic, borderline-irrational rantings of some marginal Seattle goofball who goes by an online handle that escapes me at this moment, whose schtick will wear thin within about three paragraphs for any normal person but, hey, psychotics have the right to drink wine, too, as long as it doesn’t react badly with their medication. Point being, anybody from a Nobel laureate to the guy in the tin-foil hat can find a beverage writer with whom they can vibe, IF they’re willing to look.

Robert Parker

The proper usage – and that word is very carefully chosen, because that’s what I am and Parker is and Spectator’s mass o’pinions are: Tools – for a beverage critic is as an acquaintance, a pal who gets to taste a LOT more stuff than you and with whom you have developed some history and can somewhat trust. If you read The Pour Fool and, based on my recommendations, regularly try wines, beers, and liquors that you find you don’t like, you shouldn’t be reading my reviews. I’m not trying to torpedo my readership but I want you to drink good stuff and if my tastes aren’t compatible with yours, The Pour Fool is a waste of your time and money. When I was just getting started in the Seattle wine trade, I had the late Tom Stockely as my muse, along with The Spanish Table’s uber-passionate owner, Steve Winston, both of whom proved to fit my palate like an old shoe. Later, as I branched out in all beverages, I developed a Method; one which worked beautifully for me for over two decades:

When I read about a wine that sounded interesting, back then, I would find at least three other reputable sources for ratings. These could be numbers or verbiage; preferably both. I used Parker, Tanzer, and Enthusiast, consistently, along with Wine & Spirits, and a dozen or so newspaper critics. If any three of those sources agreed that the wine was worthwhile, I usually bought it and rarely – very rarely – found a bottle that wasn’t at least drinkable.

James Suckling

So…is this my own version of crowd-sourcing? Yeah…but the crowd is made up of people with proven credentials in their fields. And that very thing is why crowd-sourcing sucks: We have ZERO guarantee that even ONE of the crowd that generated that number knows anything at all about the style of beer or wine or Whiskey reviewed or even the beverage in general. What is a 80 point rating given to an Australian Shiraz or a Pale Ale or a Canadian Whiskey worth if the rating is given by a person who doesn’t drink and explore Shiraz or Pales or Canadian? Absolutely nothing. Put in other terms, unless you happen to be a nuclear engineer, the value is about the same as your review of that snazzy new reactor down the road.

Paul Gregutt

Even within the tiny universes of the rating sites, some people’s opinions will take on a greater weight. Several of the BeerAdvocate crowd have proven very reliable to me, over the years. Tom Brush, a BA forum regular from nearby Brier, Washington, has been an invaluable second opinion for me for years but his brand of serious, thoughtful evaluation of beers is a tiny minority on that site. But it proves out my point: I learned to trust Tom Brush’s opinions. It was NOT a product of any blind faith in BeerAdvocate or some rooty belief that everyone who posts there is actually a Beer Authority and not just some bored-stiff beer drinker who’s found a convenient online social community in which to work out their snark.

Anthony Dias Blue

All of this comes back to one Ultimate Truth: Nobody has your palate, your tastes, your likes and dislikes…but you. If you make an effort to attend wine tastings, regularly, you can eventually amass a knowledge base that will at least let you buy with confidence within a range of styles without having to rely on anybody’s opinions. Let’s be brutally honest, here: I just have far more opportunity to taste wines, beers, and spirits than you do, because that’s my job. That does not, in any way, mean that my opinions are somehow better or more evolved than yours. I share the information here because it seems wasteful to only use the knowledge to sell beverages. It should do somebody some good besides me. And I hope it does. But if you find that my taste and yours simply don’t align…with regret, I’ll send you somewhere else because you deserve to drink what you like, not what you’re told is good.