Metacritic "all but obsolete" - Double Fine

COO Justin Bailey says studio hasn't seen review aggregator or traditional gaming press impact sales

Metacritic has been a contentious subject in the industry, from those who say the review aggregator has too much influence, to those who use it to make hiring decisions. But in a session at the Montreal International Game Summit today, Double Fine Productions COO Justin Bailey suggested the site doesn't actually matter all that much.

In his presentation, Bailey ran down a list of things that, in Double Fine's own experience, have and haven't made an impact on the sales of their games. On the "what works" list were Let's Play videos, Steam free weekends, and friendly informal cross-promotion with other indie developers. On the list of what doesn't work were free-app-of-the-day promotions, trade shows, and Metacritic.

"There's no need to obsess over Metacritic," Bailey said. "We've basically seen Metacritic all but become obsolete right now... We get obsessed with it. Other developers get obsessed with it. The press gets obsessed with it. But it doesn't really matter, as far as the sales of the game."

Similarly unimportant when it comes to moving the needle on sales was the traditional gaming press. Bailey said when he first arrived at Double Fine, one of his big misconceptions was that the press would have a huge impact on sales. Even with the company's Double Fine Presents publishing service, Bailey had expected that simply including these other games in Double Fine's press list would help them get a lot more traction and exposure, but that didn't pan out.

"I think who's looking at the gaming press are for the most part other developers," Bailey said. "And so it seems important to other developers. I'm just not certain how many gamers are going to the press. It seems they're being siphoned off into Let's Play and other avenues. They're going to review sites as the gateway to find their reviews."

Bailey pointed to the Disney acquisition of Maker Studios and Amazon's purchase of Twitch as evidence that larger companies have already established tremendous value for alternatives to the traditional press. Bailey said the media isn't completely useless from a financial perspective, however. When Double Fine was Kickstarting Broken Age, Bailey said the non-gaming press--particularly outlets like USA Today and Forbes--was especially helpful.

Rounding out the lists of what works and what doesn't were bundles, which appeared on both sides of the equation. Bailey said that bundles can provide liquidity in the short-term, but cautioned against their long-term impacts, suggesting developers wait until games have been out for two years or so before including them in a bundle promotion.

"I'm worried about the effect bundles will have because of the consumer expectations that I shouldn't pay more than $2," Bailey said.

More stories

Double Fine publishing boss Greg Rice departs

Vice president of business development leaves just five months after surprise Microsoft acquisition

By James Batchelor

Ooblets will now be self-published on Xbox One and PC

Co-creator Rebecca Cordingley says change will have no material effect on game

By Rebekah Valentine

Latest comments (22)

Christopher Bowen Editor in Chief, Gaming Bus7 years ago
He's right. Not only is Metacritic no longer relevant because of the number of games disproportionate to AAA releases that typically get attention there, many of the sources they draw from now are simply not trustworthy or reliable sources.
3Sign inorRegisterto rate and reply
Benjamin Crause Supervisor Central Support, Nintendo of Europe7 years ago
Finally someone said it, thanks.
2Sign inorRegisterto rate and reply
Jeremiah Moss Software Developer 7 years ago
To me, the number has never really been the highlight of a review. The reason being, a single number can't tell you if you'll find the game entertaining. Me? I like strategy and puzzles, so I tend to like those types of games. However, I have a family member that is really into shooters and loves them.

So a simple number won't say much about whether I will like the game, or whether my family member will like the game. Different people are entertained by different things.

So it's my opinion that the centerpiece of a review is really the description, where the reviewer describes the game and how it plays. It's my opinion that Metacritic gives undue weight to the idea of using a scoring system.
2Sign inorRegisterto rate and reply
Show all comments (22)
André Bernhardt Free Bird, IndieAdvisor7 years ago
The video isn´t available anywhere - or?
0Sign inorRegisterto rate and reply
James Brightman Editor, North America, GamesIndustry.biz7 years ago
Andre, I don't think a video is available. Brendan was at this session in person.

Edited 2 times. Last edit by James Brightman on 11th November 2014 9:35pm

1Sign inorRegisterto rate and reply
Thomas Dolby Project Manager / Lead Programmer, Ai Solve7 years ago
This tends to fall in line with my shopping habits at the moment. To me there seems to be two key areas, get my attention (so I know your game exists), and then sell me with some actual gameplay footage.

Good marketing ensures that I know the game exists, and also trailers and other promotions can build up hype. The deal-breaker is gameplay footage for me though. It's very rare I'll go for pre-order bonuses and I'm quite happy to wait a few days to see how consumer feedback forms around the game, and most crucially to start seeing consumers playing the games. With this approach I don't feel like I've made a bad purchase in a long time. Reviews (both official and unofficial) will have a place for a time in terms in helping sway a decision that's on the fence, but numbers have never been less relevant to me.
0Sign inorRegisterto rate and reply
Maged Hamdy Studying Computer Science, Rochester Institute of Technology7 years ago
Eh, it's good for Kickstarters and general publicity. When Kotaku is still worth 700m I'm thinking it's still SOMETHING, right?

I wonder if does not apply in this situation, as it doesn't do reviews.
0Sign inorRegisterto rate and reply
Anthony Gowland Director, Ant Workshop7 years ago
Even more so in a world where the major platforms (next gen consoles & PC) and services like Twitch make recording & streaming your gameplay so easy. It's really easy for players these days to get a good idea of what a game's really like, even when no demos are available.
1Sign inorRegisterto rate and reply
Gareth Wilson Design Director, SUMO Digital7 years ago
I'm not so sure about this, especially in the big budget AAA console market. I agree its less important for indie games, mobile games etc.
3Sign inorRegisterto rate and reply
Richard Browne Head of External Projects, Digital Extremes7 years ago
I'm with Gareth, when it comes to AAA this is nonsense. A massive marketing campaign will launch you well but without decent Metacritic the tail sales will disappear. For Indie Games or mobile its never been relevant.
0Sign inorRegisterto rate and reply
I tend to at least look at metacritics for the users overall meta score. Im not so much interested in press/experts, but users comments often prove to be insightful and many times have warned me away from poor decisions.
3Sign inorRegisterto rate and reply
Anthony Chan7 years ago
I wouldn't say Metacritic is "obsolete" because at the end of the day, there are casual gamers looking to add another game to their collection for consoles that rely on numbers. Metacritic is pretty much obsolete for PC due to Steam "evolution" and for mobile, because the respective app stores (Apple and Google Play) are excellent at pushing ratings and user reviews.

For PC, Steam being the most effective and most efficient platform for selling PC games, provides most of the information to general gamers to make a informed purchase. There are trailers, gameplay videos, user reviews, comments, etc. It effectively replaced metacritic.

For Mobile, the app stores do the exact same thing. However, I would note there seems to be more emphasis on the "star rating" system. So I actually would disagree that there is a reduced emphasis on numbers. At the end of the day, your mobile gamer (who is inherently more casual than your PC gamer) relies on star ratings to be persuaded to try your game. The Star Rating is in essence no difference than a % score or a grade.

For Console, metacritic is still relevant as it aggregates and consolidates the main source of review and rating information. Prior to internet, console players relied on gaming mags to get reviews and scores. Back then scores mattered greatly. They still do now to these players. The only difference, mags turned into websites (i.e. gamespot, ign), and now it is even quicker with metacritic ( I can see every mag's score at one go). Developers would be naive to believe that scores don't matter. Scores may not affect sales in some cases (i.e. over hyped existing franchises) but they still do matter when trying to make an informed purchase.

Regardless of the objectivity of the source of those scores (i.e. other users in mobile, gaming journalists for console, etc), scores matter. They matter just like awards do - silly thing if you ask me, but the edition that says "game of the year 20xx" pushes sales too. I agree, there may be less reliance on scores for indie, but I also wonder if that is because indies in general are not marketed to the masses - as masses rely on scores to quickly quantify justification for purchasing a game.
1Sign inorRegisterto rate and reply
David Canela Game & Audio Designer 7 years ago
this is completely off-topic, sorry, but does anyone else suddenly remember Broken Age and its 2nd act that was supposed to follow the first? whatever happened to that game?
0Sign inorRegisterto rate and reply
James Berg Games User Researcher 7 years ago
I don't think a high Metacritic guarantees you any sales, but it can certainly cost you sales if it's poor. If you check MC and see the game is rated a 65, that's a strong indicator that others didn't find the game good.
0Sign inorRegisterto rate and reply
Al Nelson Producer, Tripwire Interactive7 years ago
Ubi's 9% stock hit today is being blamed on a low Metacritic score for Assassin's Creed Unity...
3Sign inorRegisterto rate and reply
Jeremiah Moss Software Developer 7 years ago
I'm with Gareth, when it comes to AAA this is nonsense. A massive marketing campaign will launch you well but without decent Metacritic the tail sales will disappear. For Indie Games or mobile its never been relevant.
While this is maybe the way things are, I'm not entirely sure this is the way things ought to be, for the reasons I've described.

I think the most recent changes to the Steam store is showing us the future of how tail sales should be handled: Players express an interest in certain game types, and based on those interests, the store shows players games similar to their interests. This could easily include older "classic" games, extending the life of those games, and allowing players more access to games they love, both new and old.
2Sign inorRegisterto rate and reply
Rafa Ferrer Localisation Manager, Red Comet Media7 years ago
@David - As of October 17th, DF announced Tim had finished writing Act 2 and the Finale and they had already entered the studio for music and voice-over recording. (Source: The Broken Age Project Update newsletter)
0Sign inorRegisterto rate and reply
Alfonso Sexto Lead Tester, Ubisoft Germany7 years ago
Metacritic should just do a purge and remove the obviously failed "user vote" section. It's nothing but a Haters vs Fanboys troll-fest battleground
4Sign inorRegisterto rate and reply
Gary LaRochelle Digital Artist / UI/UX Designer / Game Designer, Flea Ranch Games7 years ago
@Todd-I look to user reviews as well. But when I see a bunch of five star scores with just one word in the review (and a user's name like "398gh5k8"), you pretty much know that the developer has paid a bunch of people to flood the site with phony rave reviews. In that case I just ignore the game.

Edited 1 times. Last edit by Gary LaRochelle on 13th November 2014 2:28pm

0Sign inorRegisterto rate and reply
Adam Jordan Community Management/Moderation 7 years ago
People still use Metacritic? I thought it was swept under the rug along with Gamespy and Games for Windows Live

On a serious note, I can't remember the last time I took a Metacritic score seriously or to heart but I'm guessing we're talking years and years ago now. Though unlike others, if people do use it, that's great but I would rather go with word of mouth, generally from friends.
1Sign inorRegisterto rate and reply
Keith Plesha Studio Recruiter, Raven Software7 years ago
The more important point is that the game be looked at objectively and not subjectively. Clearly someone who is more interested in FPS shooters is likely to rate an FPS shooter subjectively differently than someone who doesn't care for them. This is the beauty of user generated critique (especially on Steam). You can tell what genres the person reviewing the game spends their time on. However, liking a particular genre really only comes down to gameplay. Liking a genre doesn't make the graphics look better or the frame rate better. It also certainly doesn't affect the number of bugs or technical issues at launch. Metacritic has certainly prevented me from purchasing a game before, but I you should always use multiple metrics/sources to truly garner a well-rounded opinion. Metacritic scores do dictate business decisions, but it may have less value on player buying habits as compared to large marketing budgets.
0Sign inorRegisterto rate and reply
Jamie Knight International Editor in Chief, Playnation7 years ago
GREAT article. Never used Metacritic, never understood it ( a review of reviews? nice work if you can't do it so let someone else get it for you )

I don't support them, don''t use their format, don't allow their format on any publication I am involved with and will not even put their name in a title or tag to give them the click bait view

Maybe if they actually DID something that would change, but merely saying " everybody pay attention to me because I made a group of everyone else work " is not praise worthy and certainly not deserving of the attention

( in my humble opinion, of course )

Edited 2 times. Last edit by Jamie Knight on 19th November 2014 6:31pm

1Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.