Home

TPFThis isn’t going to be hard to write as some of what you read here, mainly because I’ve been dealing with this issue for ten solid years (almost daily) and really have no more thinking to do about it.

Crowd-sourced ratings – for anything; beer, whiskey, wine, chocolate, movies, restaurants, you name it – actually do the people they’re intended to help about as much good as they’d get from opening the yellow pages or a beverage catalog or standing in front of a store rack, closing their eyes, and picking at random. 

3e07d7b7e26d7f5b6f1557b47904b53cBefore you ask, NO, this is not the rantings of a beverage critic who’s worried about his own credibility. The day I start even thinking of how my “cred” will be affected by what I write is the day I’ll shut this puppy down and go fishin’. I was first asked about the phenomenon of crowd-sourced reviews in 2003, by a wine sales rep who stumbled upon a site where European consumers reviewed wines. I remember visiting it (the thing is LONG gone by now) and finding two things: 1) the level of writing skill in Europe is absolutely no better than it is here in the US, and 2) the Euro crowd was every bit as prone to servicing their personal stylistic agendae as Americans. EVERY US wine rated on that site was accompanied by completely dismissive verbiage. “Drinkable” was the nicest remark I found. It was SO obviously slanted toward French and Italian wines (Spain got the same trashing as the US) that, unless you were squarely in that stylistic rut yourself, it was practically useless. The same names appeared next to dozens of reviews and it was, very quickly, possible to predict exactly what they’d rate a wine, just by the name and origins.

Wine SnobberyThe lineal descendants of those fatheads populate the major rating sites of today: Yelp, RateBeer. BeerAdvocate, Epinions, and every other crowd-sourced review site. See for yourself: check how many 95 point+ reviews you find on the beer sites for any style of beer except IPA, Stout, Sour, Barleywine, Belgian Quads, Brett Ales, and one or two other stylistic niches. Check the scores for Pale Ales, Amber-Reds, ESB, Scottish or Scotch ales, Porters, Lagers in general, and Pilsners in particular. The readers and users of  these sites are almost always slaves to their personal preferences and current trends. It’s completely inarguable that nobody can reasonably rate a Pale Ale or an Amber by the same standard you’d apply to an Imperial IPA or a Flanders Red, just the same way as it’s impossible to legitimately compare the merits of a Napa Cabernet with a Washington Riesling or an Italian Sangiovese. Any of those are apples and oranges situations. And yet we’re led to believe that Pale Ales and Pilsners are lesser beers than the fawned-over Triple and Quad IPAs of the Moment. Pale Ales are their own thing and deserve to be judged versus the merits of their own style…and it’s just not done, in any general sense.

Todd Alstrom of BeerAdvocate

Todd Alstrom of BeerAdvocate

Group-think is a real thing; it’s been proven countless times in research and anecdotal studies. You get a group of six or seven (or fifteen) young men, 25 – 35, who visit breweries together on a regular basis, and there will be an Alpha Dog in the pack; maybe more than one. Those guys will set the stylistic preferences of the group and even those among them who may have not agreed with the Alphas to begin with will start to gravitate toward the common view, in order to be accepted. This also works among the forum responders on the rating websites. BeerAdvocate is notorious for its rowdy forum denizens, many of whom are firmly of the opinion that they know a LOT more about beer than the Alstrom brothers, the surprtisingly erudite siblings who own and run BA. (They’re wrong but you can’t tell ’em that.) I was ridiculed in the BeerAdvocate forums for – among other things – assigning a gaudy rating to Ninkasi “Imperiale”. One of the forumites responded with this: “You gave 99 points to that Ninkasi Imperiale crap! We already said what we think of that beer here!” The inference was clear: this person would never dream of liking a beer that was not properly anointed by his peer group, a form of lemming-like conformity which is emphatically NOT a part of my job.

Marvin Shanken, Wine Spectator publisher

Marvin Shanken, Wine Spectator publisher

I get offended messages from some of the Wine Spectator folks whenever I opine that I find their ratings fishy because of the fact that they run tasting groups. Quite aside from the indisputable fact that WS has become, as Dan Jenkins famously said of Sports Illustrated,  “a slick cookbook for your basic twoyacht family”, there is a fairly obvious uniformity to WS reviews, over the years, which seems impossible as a result of a collection of individual palates…unless they strongly influence one another. People in groups rarely tend to agree completely and yet we get Spectator reviews written by one of their number; reviews which stand as Spectator’s lasting opinion of that wine, raising the question of whose dissenting opinion did we not read. Highly erudite wine people tend to harbor certain prejudices: France is Supreme, certain styles of wine are inherently superior to others, and that inexplicable bell curve that says that wines with a sticker price of $40 and up are presumptively better than those below $20. In any group of people who evaluate anything, the goal is a majority opinion and, preferably, a uniformity of opinion. Squeaky wheels within such groups sometimes get the grease…and sometimes a tire change is the remedy.

Stephen Tanzer

Stephen Tanzer

I don’t mean to single out Wine Spectator, either. They’re just the most visible example and the one that touts their group tasting model most proudly. But in terms of how all this applies to you, the American wine consumer, I submit that it is far – as in light years far – more productive and reliable to find a like-minded individual critic, invest a bit of time to see if their palate is at all attuned to your own, and, if not, Move On. There is, as far as I know, no serious shortage of people out there who review beverages, from the uber-popular types like Anthony Dias Blue and Robert Parker and Steven Tanzer, to cranks like me and Doctor Vino. There is, literally, some common vibe to be found by everybody who’s trying to explore wine, from the gentlemanly work of my colleague, Paul Gregutt to the dyspeptic, borderline-irrational rantings of some marginal Seattle goofball who goes by an online handle that escapes me at this moment, whose schtick will wear thin within about three paragraphs for any normal person but, hey, psychotics have the right to drink wine, too, as long as it doesn’t react badly with their medication. Point being, anybody from a Nobel laureate to the guy in the tin-foil hat can find a beverage writer with whom they can vibe, IF they’re willing to look.

Robert Parker

Robert Parker

The proper usage – and that word is very carefully chosen, because that’s what I am and Parker is and Spectator’s mass o’pinions are: Tools – for a beverage critic is as an acquaintance, a pal who gets to taste a LOT more stuff than you and with whom you have developed some history and can somewhat trust. If you read The Pour Fool and, based on my recommendations, regularly try wines, beers, and liquors that you find you don’t like, you shouldn’t be reading my reviews. I’m not trying to torpedo my readership but I want you to drink good stuff and if my tastes aren’t compatible with yours, The Pour Fool is a waste of your time and money. When I was just getting started in the Seattle wine trade, I had the late Tom Stockely as my muse, along with The Spanish Table’s uber-passionate owner, Steve Winston, both of whom proved to fit my palate like an old shoe. Later, as I branched out in all beverages, I developed a Method; one which worked beautifully for me for over two decades:

When I read about a wine that sounded interesting, back then, I would find at least three other reputable sources for ratings. These could be numbers or verbiage; preferably both. I used Parker, Tanzer, and Enthusiast, consistently, along with Wine & Spirits, and a dozen or so newspaper critics. If any three of those sources agreed that the wine was worthwhile, I usually bought it and rarely – very rarely – found a bottle that wasn’t at least drinkable.

James Suckling

James Suckling

So…is this my own version of crowd-sourcing? Yeah…but the crowd is made up of people with proven credentials in their fields. And that very thing is why crowd-sourcing sucks: We have ZERO guarantee that even ONE of the crowd that generated that number knows anything at all about the style of beer or wine or Whiskey reviewed or even the beverage in general. What is a 80 point rating given to an Australian Shiraz or a Pale Ale or a Canadian Whiskey worth if the rating is given by a person who doesn’t drink and explore Shiraz or Pales or Canadian? Absolutely nothing. Put in other terms, unless you happen to be a nuclear engineer, the value is about the same as your review of that snazzy new reactor down the road.

Paul Gregutt

Paul Gregutt

Even within the tiny universes of the rating sites, some people’s opinions will take on a greater weight. Several of the BeerAdvocate crowd have proven very reliable to me, over the years. Tom Brush, a BA forum regular from nearby Brier, Washington, has been an invaluable second opinion for me for years but his brand of serious, thoughtful evaluation of beers is a tiny minority on that site. But it proves out my point: I learned to trust Tom Brush’s opinions. It was NOT a product of any blind faith in BeerAdvocate or some rooty belief that everyone who posts there is actually a Beer Authority and not just some bored-stiff beer drinker who’s found a convenient online social community in which to work out their snark.

Anthony Dias Blue

Anthony Dias Blue

All of this comes back to one Ultimate Truth: Nobody has your palate, your tastes, your likes and dislikes…but you. If you make an effort to attend wine tastings, regularly, you can eventually amass a knowledge base that will at least let you buy with confidence within a range of styles without having to rely on anybody’s opinions. Let’s be brutally honest, here: I just have far more opportunity to taste wines, beers, and spirits than you do, because that’s my job. That does not, in any way, mean that my opinions are somehow better or more evolved than yours. I share the information here because it seems wasteful to only use the knowledge to sell beverages. It should do somebody some good besides me. And I hope it does. But if you find that my taste and yours simply don’t align…with regret, I’ll send you somewhere else because you deserve to drink what you like, not what you’re told is good.

12 thoughts on “Crowd-Sourced Ratings and Why They Suck

  1. Two things
    One; Where do “we” draw the line? For example I like porter’s but I don’t like vanilla forward porter’s. Does that mean I shouldn’t score a vanilla forward porter?
    Two; More related to the Pourfool not giving bad reviews. Shouldn’t you at least mention them and say something like ‘score: sub 70?’ After all if I am looking for my palate match, of a reviewer, how do I really know if I match unless there is a low end too?

    Like

    • No, I shouldn’t mention what I would have to score low because A) I see no legitimate reason for promoting or even acknowledging mediocrity, and B) I LIKE brewers and winemakers and distillers and see no reason to take shots or negatively affect their sales effort by calling questionable products into question. Tasting them will raise all the questions that need to be raised. And, yes, if it is not your JOB to review vanilla-forward Porters, I think you should either review them and mention that prejudice or avoid reviewing them. For me, this is a JOB, not a hobby. I try, as much as possible, to take my tastes out of the process and remember that, just because I might not like Vanilla Porters, that doesn’t mean the reader doesn’t, and I have to taste 150+ Porters a year, so I learn to accept that which I don’t like and look for those virtues common to all beverages: balance, composition, textures, colors, off-flavors, and value. I want to celebrate greatness, not publicize the merely good or serviceable or the truly bad.

      Like

      • But in craft beer isn’t it the participation and sharing of opinions that is part of the appeal? To say people shouldn’t because they don’t have a certain level of knowledge is basically saying people shouldn’t share their experiences because they don’t have enough of them yet?
        C’mon, reviews are opinions, not facts. Sure, some are better than most, and yours are very good, but to say crowd sourced ratings suck is a but too much of ‘get off my lawn’.

        Like

        • I didn’t say that the people shouldn’t write their reviews. They can do what they like with their time. What I’m saying is that anyone with any sense at all won’t give those ratings a lot of credence. If we could all trust crowd reviewers to give ratings ONLY to those styles of beer they know and understand, they might be more valid. But I know for an absolute fact that there are a lot of people who review beers on RB and BA who only drink IPAs but then write about Pale Ales and Ambers and other types of less aggressive beers and down-rate them for being “flabby” or “too mild” – in short, judging those very different styles of beer by the parameters of an IPA. I know this for a fact because I get emails from these folks and know a couple dozen of them personally, here in Seattle. Several have cheerfully admitted that they drink nothing but IPAs and defended their 82 point scores (or, in BA’s ratings, roughly 3.6) by reasoning that, because most people they know prefer IPAs, other styles of beer need to measure up or rightly be rated lowly. This is so freakin’ WRONG, on so many levels, that it almost makes me go cross-eyed but you can’t tell these meatheads anything. They all think that they and their five or six buddies are the Ultimate Beer Authorities. This being America, however, they have every right to post as much illogic as they like on RB and BA. But the rest of us do not and SHOULD NOT put any stock in those amalgamated ratings. I don’t dispute anyone’s right to rate stuff. What I KNOW is that those ratings have no more real substance and credibility than choosing by flipping a coin and that people who put any trust in them are just begging to be misled.

          Liked by 1 person

          • I’m glad I took the time to comment and thanks for taking the time to respond. We are more on the same page than I ever thought. I completely agree that people that only drink IPAs and only rate face meters a 100 should not be critical of other styles. I put very little stock in the number on a rating and focus more on reviews with words – and there hopefully someone can explain there are new to a style or that something just isn’t their thing. Re-reading the post after your reply puts it in a bit of different context – or maybe it was because I was originally reading it on my smartphone screen. Thanks again for taking the time to reply.
            Based on your top 25 list I came across a few minutes ago, our tastes appear pretty similar. Looking forward to more reviews.

            Like

  2. Beergraphs has a stat called style+ that takes the crowd sourced ratings and compares them only to other ratings in that style. DIPAs and barrel aged stouts still dominate the overall leaderboards but you can at least sort the rating to compare within a style and see how one pils matches up against all pils, for example.

    Like

    • And beergraphs has a readership that’s even smaller than this blog’s. I like the way they do things but it, sadly, doesn’t reflect the vast majority of beer ratings. Until that happens – we can only hope! – what I said in this post represents my views on ALL crowd-sourced ratings.

      BTW, my FB pal, Joseph Tucker, owner of RateBeer swears that they have an algorithm that accounts and adjusts for trendiness, supposedly balancing out the current mania for DIPA, Barrel-Aged, and Sour/Brett. The problem is that those categories you mentioned are just about the only ones that most of the participants in both major sites ever seriously drink and evaluate. Most of those people will drink a Pale Ale or an Amber/Red or an ESB only if they’re handed one at a party and then only if there’s no IPA handy. I don’t care how finely tuned your algorithm is, the beers that are not currently hot simply receive lower scores than those that are and that is even true on beergraphs.

      Like

Speak yer piece, Pilgrim.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s