Sections

Are review scores pointless?

Eurogamer is the latest to abandon the "out of 10" ratings system, but what do developers think of the idea?

Earlier today, GamesIndustry.biz sister site Eurogamer announced that it is dropping scores from its reviews process. In place of its previous 1-10 scoring system, Eurogamer will henceforth have only three ratings: "recommended," "essential," or simply "avoid."

It's not an unprecedented move by any stretch, but it is another indication that the games media is moving away from the traditional review format. Just weeks before it found itself shut down for good, Joystiq made headlines by dropping its own review scores, bringing it more in line with a policy longtime rival Kotaku had espoused for years.

Clearly, the game review format as it existed for decades is an ill fit for an industry where buggy console games can be patched in short order, where the games-as-a-service approach, if correctly applied, ensures that a game will never be worse than on the day it launches and reviews hit. But is dropping review scores the answer? When Joystiq announced its policy change in January, I asked a handful of people throughout the industry for their first reactions, some of which are included below.

Brian Fargo, inXile Entertainment founder

"My understanding is that websites either have a policy of not updating the scores based on future editions or Metacritic won't update their rating even if they do. With that in mind I would prefer that gaming sites use a summary system and let the players give it a score if they want. Games are more of a service than in the '90s and certainly more complex. The updated versions of games could be light years ahead of the original release and that needs to get reflected somehow. Developers should be rewarded for continuing to support their games long after launch and this goes a step towards that."

Nathan Vella, Capy Games president

"I grew up with review scores. They're etched in my brain as a metric for 'quality' because of how much of my life (as a player and a developer) has been spent looking at them in magazines, and later, online. However, I am totally aware that it is my history with them that drives my comfort with them. I think this history affects a lot of us developers, consciously or subconsciously."

Rami Ismail, Vlambeer co-founder

"I've always disliked the notion of scores on something as abstract and subjective as games. This is the one medium in which both creator and user express creativity, in which the player is tangible part of the experience - and then magically a number appears based on someone's experience. I'm fine with them existing, I'm fine with them not existing."

Perrin Kaplan, Zebra Partners principal, former Nintendo of America VP of marketing and corporate affairs

"The technical measurement plus the dynamics of the marketplace are so different now. With games from consoles to Steam and mobile, it is apples and oranges at best. That said, for developers, business leaders, marketers, those ratings have meant a lot and in many cases, still do."

Paul Hellquist, Robot Entertainment lead designer

"Hallelujah! So many sites have so many different models and ways of trying to project their opinions and they're so dramatically different, it just really skews people's perceptions because they go to the scores without actually reading the text. So I love it. I've always liked Kotaku's play-it-or-don't-play-it system. Siskel and Ebert always had it right. Thumbs up, go see it. Thumbs down, don't waste your time. That's what it comes down to, so I'm excited to see more sites go to a system like that. Read the text and decide if what we're describing sounds like something you'd enjoy."

Patrick Hudson, Robot Entertainment CEO

"Isn't Destiny a pretty good case study of this recently? How much wrangling and articles have been written about their Metacritic score? Does it matter if they're a 70-something? There are millions and millions of people playing and enjoying that game every night. And I think it becomes harder in a world of live games that grow over a long period of time. They're going to be pumping massive content into Destiny for years to come; is it fair that they get saddled with some review score from three years ago? It's a real problem."

A reviewer's take

I also reached out to a former co-worker in ex-GameSpot editor-in-chief Justin Calvert. At this point, Calvert has left the gaming media entirely, but over his near decade-and-a-half with the site--much of it spent as a staff reviewer--he said the things publishers wanted from reviews systems were essentially the same.

"High scores and box quotes mostly, but then the vast majority of my dealings with publishers and developers were through PR channels, so that's hardly surprising," Calvert said. "I'll say that when dealing with development teams directly they were often genuinely interested in discussing and taking on board feedback from reviewers."

That's not to say the way publishers treat reviews has stayed the same. For one thing, Calvert said publishers have become a lot less concerned about having reviews hit before launch to build up hype.

"They're less dependent on reviews for preorders and day-one sales because they've gotten so much better at speaking to their audiences directly," Calvert said. "The impression I got, at least from some companies, was that the potential benefits of a positive review pre-launch were outweighed by the damage that a negative review could do at the end of a successful marketing campaign.

Calvert himself is still a fan of systems with review scores, provided they're presented with context such as when the review was published, and which version of the game was reviewed.

"I've been basing my own game-purchasing decision on reviews ever since I picked up the first issue of Zzap! 64 magazine in the UK almost 30 years ago, so I hope that they continue to be relevant in the future," Calvert said. "I don't put nearly as much stock in user ratings where video games are concerned as I do when buying practically anything else (thick wool socks on Amazon being a recent example); I'm not entirely sure why that is, but after working alongside so many great reviewers during my 14 years at GameSpot I'm still inclined to look to professional reviewers first. The day that I'm unable to go online and read a Kevin VanOrd review of New Game X will be a sad day indeed."

That said, even Calvert admits there are times when YouTube videos or Twitch streams are the basis for his purchasing decisions.

"There's something very appealing about watching a game being played and knowing that the footage hasn't been edited in a way that might misrepresent the experience," Calvert said.

More stories

Leading names from YouTube, TV, film, books and games set for Changing Channels

The new event in our GamesIndustry.biz Live series takes place on August 13th for free

By Christopher Dring

Tencent reportedly driving Huya and DouYu to merge into livestreaming giant

Merging Chinese Twitch rivals would create a business worth $10 billion with 300 million users

By James Batchelor

Latest comments (13)

Ron Dippold Software/Firmware Engineer 5 years ago
Most of the arguments are about whether a single score adequately represents the game (probably not), whether any game can be considered done at launch any more (probably not), how you can reasonably compare similar numbers (you can't), and what an 8.0 actually means (ballpark).

But industry people often miss the most useful purpose of review scores for the reader - signaling whether they should read the review or not. I see 'just read the review' here too but nobody's got time, or even the desire, to read all the reviews for all the games, or even most of them. That's unpleasant to contemplate as the person who wrote the review or as the dev who put your heart into the game, but that's how it is. If there's a game you're interested in you might read the review. Otherwise you check out the score and if it's a 9 or 10 you read to see what's so great about it - maybe you should be interested! If it's a 1-3 you read for the trainwreck.

You can do without the score - a good title or lede could serve the same purpose. Kotaku's review setup doesn't adequately address this; I guess their readership is large enough they don't care. Eurogamer's 'essential' and 'avoid' might do, though they lack the punch. I do actually read a lot of reviews, and for most mags or sites this just isn't adequately conveyed without the score. Edge, for example, goes out of their way to avoid giving you any hints in the early text - that's just one of their conceits, but it leads to a lot of people admitting they just go to the end to check the score first.

Edited 2 times. Last edit by Ron Dippold on 10th February 2015 6:11pm

1Sign inorRegisterto rate and reply
Jordan Lund Columnist 5 years ago
The gold standard for reviews for decades was Siskel and Ebert and they got along fine with "Thumbs Up" and "Thumbs Down". This would be the equivalent of Essential (2 thumbs up), Recommended (1 up, 1 down) and Avoid (2 thumbs down).

I don't have a problem with this. The only trick is, and perhaps the point, is that it makes it harder for aggregators. How is Metacritic going to compile Essential, Recommended and Avoid? 100%, 50%, 0%? Or will they just ignore such sites all together? It's already hard enough compiling sites that are 10 based vs. 5 based. 4/5 is a good game, but 80% is barely above average. A 3 tier scale would make it more confusing at an aggregate level.
2Sign inorRegisterto rate and reply
Kieren Bloomfield Software Engineer, EA Sports5 years ago
Bingo Ron! They've just removed the mediocre/OK score since they fear they are the types of reviews that aren't getting read. They haven't actually removed a scoring system at all, they've just trimmed it down to three and I think there's a vast gap between 'recommended' and 'avoid' in most peoples eyes. oh right, that's the bit where I'm meant to read the text!
1Sign inorRegisterto rate and reply
Show all comments (13)
Christopher Bowen Editor in Chief, Gaming Bus5 years ago
I can say one thing: as a writer, I don't care if taking out review scores makes publishers mad. In fact, I'm all for anything that shifts that leverage a little more towards us.

EDIT: I also could care less about what the aggregators think, who do none of the work and yet make all of the benefit. Aggregators like Metacritic are a problem specifically because it becomes in effect a race to the fringes.

Edited 2 times. Last edit by Christopher Bowen on 10th February 2015 7:30pm

1Sign inorRegisterto rate and reply
Greg Wilcox Creator, Destroy All Fanboys! 5 years ago
Aggregation is pretty much one thing that needs to die a quick, painful digital death. It's been swirling in the bowl ever since scoring was opened up to spiteful idiots not in the field who do nothing but slam games down just for "fun". Meanwhile, the more clueless users of aggregate sites use THOSE "reviews" combined with whatever lower scores a game gets from actual critics to judge the product. And that's usually before these people touch a controller. Or worse, perhaps they've played a demo and hated it, but use that as a basis to grab the lowest review scores and justify scribbling up a low score post of their own.

In the meantime, some decent reviews done by people who "get" these games and whip up some well-done constructive criticism get derided by some who think a good review now means a writer was "paid off" by a publisher. Ugh.

It turns out that a lot of these fine folk don't even READ full reviews anymore. They look at the score, scan for negatives in the review and run straight to a message board to paste what they've cut and claim a sort of "victory" over something. What? their ignorance about context and how to turn a favorable (but average to good) score into a "don't buy this game!" post? Yeesh.
4Sign inorRegisterto rate and reply
Chris Payne Managing Director & Founder, Quantum Soup Studios5 years ago
I'll be glad to see the back of scores. It's been a while since anyone's used the frankly ridiculous percentile ratings we used to see, and scores out of ten aren't much better. In the end any scoring system boils down to whether the reviewer liked the game or not, and the sooner we acknowledge that subjectivity, the sooner we can start paying attention to the tastes of the reviewers and choose to read the sites whose opinions best match our own. That's a far better reference point than an aggregate score that by definition trends towards average.

[EDIT] Just remembered that back in the late 80's/early 90's ACE Magazine used to score games out of 1000 - with a straight face too :)

Edited 1 times. Last edit by Chris Payne on 11th February 2015 12:59pm

3Sign inorRegisterto rate and reply
Patrick Walker VP, Insights, EEDAR5 years ago
If I'm an industry executive trying to make a profit review scores certainly matter to me.

http://www.gamesindustry.biz/articles/2014-12-11-the-data-is-in-and-your-review-score-still-matters-eedar
0Sign inorRegisterto rate and reply
Eyal Teler Programmer 5 years ago
I'm with Ron Dippold on this. As a player I want scores. They're the easiest way to focus my attention on a game. I also want them aggregated, so that I can see in one place how good multiple publications thought a game was.

When it comes to changes over time, removing scores hurts that. If I see a list of reviews with a score of 5 and then one 8 review, that one would stand out and I might check it out to see why it thought differently. If that was because the game got patched and improved considerably, then I will have a new perspective on the game. Without scores, and without having a list of them on one page, I will have no way of finding out that a game improved.
0Sign inorRegisterto rate and reply
Marty Howe Director, Figurehead Studios5 years ago
I always read Metacritic reviews, just out of curiosity (not to enforce a decision) I always read the bad reviews, never the good ones. I mean bad reviews, like a score of 0, or 10, or 20. Because I want to know what the negatives are, immediately.

The reviews that are 80, 90+ I am cautious, because I know the reviewer is a friend of the game company, had free lunch, plane tickets, hotels etc. So I'm a bit wary.

The Metacritic user reviews are the best though, from actual consumers (players) They're honest, not afraid to hurt anyone's feelings, and tell it like it is (including profanity ;)) because they are the ones handing over their hard earned money.
0Sign inorRegisterto rate and reply
Gary LaRochelle Digital Artist / UI/UX Designer / Game Designer, Flea Ranch Games5 years ago
The Metacritic user reviews are the best though, from actual consumers (players) They're honest, not afraid to hurt anyone's feelings, and tell it like it is (including profanity ;)) because they are the ones handing over their hard earned money.
I have to respectfully disagree with this. Because some studios pay their employees to flood the reviews with positive fake reviews. If you see a review from fh93u8vnnx and they give the game a positive one word review, you pretty much know it's a phony review that was posted just to drive up the average review score.

Edited 1 times. Last edit by Gary LaRochelle on 11th February 2015 6:24pm

2Sign inorRegisterto rate and reply
Dave Herod Senior Programmer, Codemasters5 years ago
The Metacritic user reviews are the best though, from actual consumers (players) They're honest, not afraid to hurt anyone's feelings, and tell it like it is (including profanity ;)) because they are the ones handing over their hard earned money.
Uh, what? My experience of reading the user scores is that they're usually either 10 or 0. Usually with some accompanying text explaining how there was one single thing about the game that they didn't like and therefore the entire game is an abomination that deserves no credit whatsoever.
4Sign inorRegisterto rate and reply
Klaus Preisinger Freelance Writing 5 years ago
Consider all the work going into a review, from acquisition of the game, to playing it, to writing the review. Then consider funneling your audience towards reading one number at the end and maybe the final paragraph above it. Meanwhile publishers try to influence the number and readers complain about review scores being bought. Not to speak of the fact that all people involved in this generally think Nintendo makes even less sense. Classic madness.
0Sign inorRegisterto rate and reply
Sandy Lobban Founder, Noise Me Up5 years ago
Simply too much content to cover these days, but if you have a handful of console developers who are still willng to spend 100k for a background image on your review site, then you're better to keep them sweet by reserving judgement on their titles.

Edited 2 times. Last edit by Sandy Lobban on 11th February 2015 4:57pm

1Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.