Are review scores pointless?

Earlier today, GamesIndustry.biz sister site Eurogamer announced that it is dropping scores from its reviews process. In place of its previous 1-10 scoring system, Eurogamer will henceforth have only three ratings: "recommended," "essential," or simply "avoid."

It's not an unprecedented move by any stretch, but it is another indication that the games media is moving away from the traditional review format. Just weeks before it found itself shut down for good, Joystiq made headlines by dropping its own review scores, bringing it more in line with a policy longtime rival Kotaku had espoused for years.

Clearly, the game review format as it existed for decades is an ill fit for an industry where buggy console games can be patched in short order, where the games-as-a-service approach, if correctly applied, ensures that a game will never be worse than on the day it launches and reviews hit. But is dropping review scores the answer? When Joystiq announced its policy change in January, I asked a handful of people throughout the industry for their first reactions, some of which are included below.

Brian Fargo, inXile Entertainment founder

"My understanding is that websites either have a policy of not updating the scores based on future editions or Metacritic won't update their rating even if they do. With that in mind I would prefer that gaming sites use a summary system and let the players give it a score if they want. Games are more of a service than in the '90s and certainly more complex. The updated versions of games could be light years ahead of the original release and that needs to get reflected somehow. Developers should be rewarded for continuing to support their games long after launch and this goes a step towards that."

Nathan Vella, Capy Games president

"I grew up with review scores. They're etched in my brain as a metric for 'quality' because of how much of my life (as a player and a developer) has been spent looking at them in magazines, and later, online. However, I am totally aware that it is my history with them that drives my comfort with them. I think this history affects a lot of us developers, consciously or subconsciously."

Rami Ismail, Vlambeer co-founder

"I've always disliked the notion of scores on something as abstract and subjective as games. This is the one medium in which both creator and user express creativity, in which the player is tangible part of the experience - and then magically a number appears based on someone's experience. I'm fine with them existing, I'm fine with them not existing."

Perrin Kaplan, Zebra Partners principal, former Nintendo of America VP of marketing and corporate affairs

"The technical measurement plus the dynamics of the marketplace are so different now. With games from consoles to Steam and mobile, it is apples and oranges at best. That said, for developers, business leaders, marketers, those ratings have meant a lot and in many cases, still do."

Paul Hellquist, Robot Entertainment lead designer

"Hallelujah! So many sites have so many different models and ways of trying to project their opinions and they're so dramatically different, it just really skews people's perceptions because they go to the scores without actually reading the text. So I love it. I've always liked Kotaku's play-it-or-don't-play-it system. Siskel and Ebert always had it right. Thumbs up, go see it. Thumbs down, don't waste your time. That's what it comes down to, so I'm excited to see more sites go to a system like that. Read the text and decide if what we're describing sounds like something you'd enjoy."

Patrick Hudson, Robot Entertainment CEO

"Isn't Destiny a pretty good case study of this recently? How much wrangling and articles have been written about their Metacritic score? Does it matter if they're a 70-something? There are millions and millions of people playing and enjoying that game every night. And I think it becomes harder in a world of live games that grow over a long period of time. They're going to be pumping massive content into Destiny for years to come; is it fair that they get saddled with some review score from three years ago? It's a real problem."

A reviewer's take

I also reached out to a former co-worker in ex-GameSpot editor-in-chief Justin Calvert. At this point, Calvert has left the gaming media entirely, but over his near decade-and-a-half with the site--much of it spent as a staff reviewer--he said the things publishers wanted from reviews systems were essentially the same.

"High scores and box quotes mostly, but then the vast majority of my dealings with publishers and developers were through PR channels, so that's hardly surprising," Calvert said. "I'll say that when dealing with development teams directly they were often genuinely interested in discussing and taking on board feedback from reviewers."

That's not to say the way publishers treat reviews has stayed the same. For one thing, Calvert said publishers have become a lot less concerned about having reviews hit before launch to build up hype.

"They're less dependent on reviews for preorders and day-one sales because they've gotten so much better at speaking to their audiences directly," Calvert said. "The impression I got, at least from some companies, was that the potential benefits of a positive review pre-launch were outweighed by the damage that a negative review could do at the end of a successful marketing campaign.

Calvert himself is still a fan of systems with review scores, provided they're presented with context such as when the review was published, and which version of the game was reviewed.

"I've been basing my own game-purchasing decision on reviews ever since I picked up the first issue of Zzap! 64 magazine in the UK almost 30 years ago, so I hope that they continue to be relevant in the future," Calvert said. "I don't put nearly as much stock in user ratings where video games are concerned as I do when buying practically anything else (thick wool socks on Amazon being a recent example); I'm not entirely sure why that is, but after working alongside so many great reviewers during my 14 years at GameSpot I'm still inclined to look to professional reviewers first. The day that I'm unable to go online and read a Kevin VanOrd review of New Game X will be a sad day indeed."

That said, even Calvert admits there are times when YouTube videos or Twitch streams are the basis for his purchasing decisions.

"There's something very appealing about watching a game being played and knowing that the footage hasn't been edited in a way that might misrepresent the experience," Calvert said.