Wedgwood: Metacritic pressure on devs is "ridiculous"

Splash Damage boss would prefer 5-star system for videogames ratings

Splash Damage studio director Paul Wedgwood has criticised the pressure that some developers are under to achieve specific high ratings based on Metacritic scores, particularly when financial rewards depend on it.

And he feels that the ratings system employed by some consumer videogames sites doesn't necessarily help, stating his preference for a five-star system, rather than a percentile-based one.

Speaking to late last year, Wedgwood was asked if he felt that trusting a developer's bonus on the thoughts of a selection of journalists - and specifically one instance where a team was required to turn in nine-out-of-ten scores rather than eights - wasn't a bit arbitrary.

"Personally I think it's ridiculous," he responded. "In the film industry, four stars is an amazing score. I think it's a really good idea for a developer to go to a publisher and demand that they get an additional bonus for achieving a certain review score, but it shouldn't affect their royalties or anything else. If you have a high-selling game, you have a high-selling game.

"We know that some websites score quite high and some quite low, but in general, all websites tend to score between 60 and 100. There's never a 37. It's as if that whole section doesn't exist, so zero starts at 60, so three stars, and goes up to five. It's just not really an accurate enough measure.

"I think that if anything, the games press should take the pressure off themselves, and just go across to star ratings, which for films is nothing more than a recommendation that you buy it, watch it when you get the chance, or rush out and see it straight away, and it's your personal recommendation. It's not a 'score'. If that was all you did, nobody would hate you guys for it.

"Out of ten is a good start. Percentiles put too much pressure on a journalist to justify an exact score. It puts too much pressure on the developer to try and identify these criteria that lead to very specific point increases or decreases, which is not at all what the developer should be focusing on."

He was also clear, however, that there was no such pressure on his team, now working under a deal with Bethesda owner ZeniMax Media - but freely admitted that high ratings was something that Splash Damage were keen to receive.

"Yes," he agreed,"The shameless pursuit of critical acclaim. It's not so much that it's purely the reception that we get from critics alone - we also mean critical acclaim from fans, and feeling like we've made something that they want to play. But really it's this notion that you focus on the quality of the game, you don't have a set release date irrespective of that.

"It's a sad fact that in 2007, two of the highest-selling videogames were also two of the lowest-rated. So that's one of the things we're trying to get across to students - that it's better to be a play-tester at Epic than it is a lead artist at a terribly naff studio that's focused purely on getting something out in time."

The full interview with Paul Wedgwood, in which he also talks about the company's relationship with ZeniMax, as well as the future of digital distribution, is available now.

More stories

How Tencent gave Splash Damage the "cheat codes for game development"

Expansion, acquisitions, and more original IP: CEO Richard Jolly opens up about the opportunities Tencent has created

By Marie Dealessandri

Splash Damage exploring friendlier territory with Outcasters

Veteran developer of competitive online shooters aims for approachability with its new game for Stadia

By Brendan Sinclair

Latest comments (2)

Jim Webb Executive Editor/Community Director, E-mpire Ltd. Co.13 years ago
I remember when Fox Interactive announced it planned to payout bonuses based on review scores. At first it seemed a decent idea to reward based on quality. Then I noticed a huge disparity in review scores for a certain Fox Interactive title where as most reviews gave very low scores (regardless of scoring metric used) but one or two gave incredibly high scores. That started the dirty secret of paying off reviewers.

With todays review standards, I think it's a very bad idea. As noted in the article, review scores seem to start at 60 instead of 0 as they should. 50 should mean an average title but it doesn't for most media outlets. This is thanks to the race of media outlets trying to gain favor with publishers by upping their average scores slightly above the competition. This is an industry problem that will likely never be addressed without the help of some form of media reviewers association to help formulate guidelines.

Wedgwood should not lay the blame on the scoring metrics of the media as the problem. A 100 point scale is fine when properly implemented. Rather the bonuses should be payed based on different values, not just average Metacritic or Game Rankings review scores.

Let's not forget the unfortunate notion that certain genres score better than others simply because of the affinities of most reviewers. They're more likely to rate a FPS in the 90's than a quality title geared toward a younger audience or girls. That's betting on a single horse that can't always be trusted and can sometimes be bought.
0Sign inorRegisterto rate and reply
There are other problems with review scores and their relation to payments in general right now.

I recently showed an experienced journalist a budget game designed specifically for children who told me that the game would receive 4 out of 10 on a regular games website, but 7 out of 10 on a website made specifically for children's games. Luckily the PR department of the publisher were already wise to this and had decided to market to specialised childrens press only and they did not send the game to any gaming websites.

To back this up I cna site another budget children's game on Wii and DS that I have been involved in recently. The same game scored an appauling 18% and 23% on a regular games website and a fantastic 80% and 84% on a website specializing in childrens gaming. Interestingly the specialist children's site bothered to pick up on the fact that at it's bargain price the game was relatively good value for money and appropriate for the young children it was targetted at. Whilst the mainstream website was more interesting in comparng the game to titles that have had 20 times more money put into the development, whilst at the same time overlooking the facts that:
a. the game was half the price of regular games
b. license holders limitations placed on brand representation meant that it deliberately did not include the more violence action they felt was missing.
c. it was targetted at children and not the mass market.

This illustrates how the subjective, random opinions of journalists could severely affect the income of developers under review bonus clauses (thankfully this title was not under such a clause.

The point is that it is a journalist's job to know his readers and to write his reviews accordingly, so any developer not working on a mass market game is going to find it almost impossible to garner a high enough percentage of good reviews, across a broad spectrum of online review sites, to trigger the payment of their review bonus.

I personally know of very few cases where developers actually have received their bonus money from review scores.

Another problem is that the % marks needed to trigger the bonus are often not commensurate with the development budget given to develop the game. Naturally many of games reviewers do not really stop to think that some of the titles they are reviewing (on the XBox for example) have been developed on 1% of the budget of other bigger titles such as Halo 3. Nor is their any reason that the journalists should even bother to consider this (unless the price point is lower). However if the publisher is looking for VFM from the developer then when using the review scores model for bonus payment one could argue the following case:
(i) If a game with a dev budget of $20M behind it and a retail price of $39.99 should score 90% to fulfill the bonus criteria ($222,222 invested for every 1%)
(ii) Then a game developed on a budget of $450k with a retail price of $19.99 should have to score just 2% to equal this in VFM terms... however allowing for the lower retail price (ie you need to sell 2 copies of this second game to generate the money of the first) I think it would be fairer to re-adjust this target to level the playing field... so an average review of 4% or more should trigger the bonus)

Of course what I have just proposed is ridiculous, but then the whole system of bonuses for review scores is ridiculous.... as here are too many factors out of the developers control affecing the review scores.

I much prefer good old fashioned royalties, they are easier for everyone (provided the publishers accounts department is honest and competent) and they are actually fairer for both publisher abd developer alike. There are 3 basic variants of roylaties.

a. development fee paid separately and in addition to royalties

b. development fee as an advance against royalties

c. development fee paid separately and in addition to delayed royalties
(in this case publisher can recoup an agreed sum from net receipts (usually 2 or 3 times dev cost) before royalties are paid out to developer)

All of the above formulae are usually preferable to review bonuses unless you have a budget of at least $5M to play with for development with which to make a semi competitive product.

0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.