Close
Report Comment to a Moderator Our Moderators review all comments for abusive and offensive language, and ensure comments are from Verified Users only.
Please report a comment only if you feel it requires our urgent attention.
I understand, report it. Cancel

The problem with Metacritic

The problem with Metacritic

Tue 10 Jul 2012 3:36pm GMT / 11:36am EDT / 8:36am PDT
Media

Devs and journalists debate influence of the score aggregator at Develop 2012

In a Develop panel titled Metacritic Killed the Video Game Score Splash Damage's Paul Wedgewood, Mastertronic's Andy Payne, Guardian writers Simon Parkin and Keith Stuart and ex-Future Publishing boss and PCGamesN founder James Binns all came out against the score aggregation site.

"Metacritic is Metacritic and nothing we say or do is going to change its role or change its presence or purpose or anything else," said Wedgewood early on.

"The truth is that where journalists really help is with discovery, particularly now we have this fantastic rise of indie gaming again and games that have much lower advertising budget, they don't have the opportunity to be discovered, I can rely on journalists to spend all their time doing that thing so they can find that for me."

He described it as an incredibly important role, but that ultimately what mattered was the opinion and quality criticism, the score wasn't important.

In a tense discussion much of the argument settled on the use of the numeric score, which Payne called "leading the witness," and Parkin suggested made scores seem like something scientific and empiric, when in fact they're entirely subjective.

"It gives the implication that there's some maths behind this, and that's why they go into spreadsheets, because there's some mathematical thing going on here," he said.

"When really scores are basically signifiers of how good you think the game is. It's not like a toaster or a camera."

Binns pointed out that trade media list Metacritic scores, and Steam even had them embedded into its store, when actually what consumers want is opinions from reviewers they know and trust. He also warned the audience, more than once, about the dangers of trusting early review scores.

"Always be suspicious of the first review that goes up, because somebody just wants to get the Google news result and get some cheap page views," he said.

He later added to that when asked about the publisher practice of blacklisting reviews and publications because a low score had dragged down their Metacritic average.

"The thing that is interesting is how a Metacritic score will change over time, if you look at the first reviews that go up generally they're fairly positive because friendly, smaller sites that are just are just excited to get a copy of the code may go early with a good review. And if you mapped it over time you'd see something more interesting than just journalists who are on message."

Stuart added that the biggest problem caused by Metacritic was the commodification of the relationship between the reviewer and the game and the reviewer and the industry.

"There's certainly times that I've been in trouble or writers for the Guardian have been in trouble because we've marked lower than lots of other people, and you get the developers on the phone complaining because you've drawn their Metacritic score down below a threshold where they were going to get a bonus."

He described it as troubling for reviewers to see those consequences, and suggested that inexperienced reviewers looked at Metacritic before deciding on a non-controversial score.

Suggestions to solve the problem included switching to a five star system, or, said Binns, using the other metrics available from games today to make judgement on a game's quality, like how many hours are people playing it for. Not everyone agreed.

"If you used that metric then Farmville would be the greatest game of all time," interjected Parkin.

Binns countered that on a business level, Farmville is a great game, leading Parkin to call him a monster.

"That's why critics are still important," he said. "Because they're judging it through a slightly different lens than just 'is this going to be profitable?'"

3 Comments

The inherent problem of Metacritic is its not reflective of the true meta analyses employed in scientific publications, and the sample pool size is just not significantly large enough to be proven significant. but its probably a marketing depts wet dream as teh scores can be skewed to some extent accordingly with the relative to the charm offensive and wine/dine budget.

In fact, you could have some recent games with high meta critic scores but to the average player...just floppy at day one, week one and the long tail end of sales.

Posted:2 years ago

#1

Dan Howdle Head of Content, Existent

280 810 2.9
The main point to be remembered here is that it is indeed not a scientific process, it is an opinion and with a sufficient check of reality, one that makes it easy to see that you can no more give a game a number than you can a system of government or a glass of water.

Posted:2 years ago

#2

Justin Biddle Software Developer

159 484 3.0
While equally I agree it can cause problems is there also not some truth in the fact that if a game gets slated in pretty much every review then Metacritic reveals this rather than just leaving you wondering if a particular reviewer didn't like the game. Which I'm sure that as much as the issues above are valid, many publishers would rather this information wasn't available either.

Posted:2 years ago

#3

Login or register to post

Take part in the GamesIndustry community

Register now