Everyone's a Critic
Metacritic's Marc Doyle on the career score controversy, responsible reviewing and credit where it's due
Although not a new feature, Metacritic this week disabled career scores, admitting that details used to calculate the stats weren't thorough enough, amongst complaints from developers upset that their work was either being scrutinised unfairly or ignored completely.
In this exclusive interview with GamesIndustry.biz co-founder and games editor Marc Doyle discusses the thinking behind individual scores and the process it adopted, Metacritic's influence on developers, publishers and journalists, and whether companies are purposefully withholding information about game credits.
I guess I was surprised at the attention so late in the game, because we added that feature in August when we re-launched the site. And I've gotten a few e-mails here and there, "Hey, I've done more than just these five games how can I add more information?" or that type of thing. But those e-mails were certainly not negative.
Then I think two or three high profile people in the game development industry took notice and they put it out there on Twitter and then that got going like wildfire. I just don't think too many people even noticed it was there, and so that kind of called attention to it and then through that process it became clear that if we're not able to do a comprehensive job in adding credits to individual's names then it's not right.
Like if they've worked on 30 games and we can only show four and then we take on this score, which is really just an average of those games in our database for them, then that's not fair. And we discussed it as a team and it made sense to just drop that overall number whilst still trying to build this database which will be difficult, but we're going to give it a shot. It's needless to put that number on it though.
If we're not able to do a comprehensive job in adding credits to individual's names then it's not right.
The first sort of complaint was simply, "How could you do this in such an incomplete way?" And I think a lot of people grant that it's tough, it's not as easy as IMDB getting the credits or the cast list from a studio. The industry's view of game credits... they're not as forthright about it as movie studios are, and so it's a little bit more of a challenge. Obviously other sites have taken a stab at it, with varying degrees of success.
But the other thing is the industry sees Metacritic as this gauge of quality and it's used in various forms. I know that in dealings between publishers and developers Metacritic scores come up and I'm sure that some people are more comfortable about that than others, and some are downright uncomfortable about that - and I can understand that.
I think then when it came down to the very personal nature of, "Wow, I was just a level designer on this" or "I only had a very small role in this particular game and yet I'm tagged with it". I can see where that can be troublesome to certain people. Some people even said, "Wow, I wonder if one day on my résumé companies are going to demand that I have my Metacritic career score." That is certainly not what we contemplated with those game credits.
Let's say you love a particular movie and you want to see who's behind it. You click on the director and you see what else he worked on, or you love this actor - you've never heard of her before - let's see whatever movies she's done. By the same token, you go over to games and "this was incredible", or "I love this particular voice actor", let me go to the credits page and see who these characters were. And then if you click on this person, "Oh, this person also worked on these other 10 games." It's just a way to find other products that they might like to play.
Because that's ultimately the goal of Metacritic: "What should I watch, what should I play?" Not necessarily to fuel some larger discussion over what person is more worthy than some other person. Throughout this process we've never ranked individuals, there's a certain context where we say: "Here are the top 10 games of the year by metascore", but even with that old career score - which we've now gotten rid of - we were never going to say: "Here's the top 10 level designers of all time on Metacritic." It was merely a way to collect this data and just take an average of the individual credits.