Sections

Metacritic disables developer career scores

Site admits methodology is "not nearly as comprehensive as it needs to be" but is still committed to ratings

Metacritic has pulled controversial ratings from its site that scored individual developers out of 100 based on their contributions to games.

The site collates aggregate scores for games and reviews sites, and this latest move was in conjunction with information gathered by sister site Gamefaqs.com - but it admitted information was not as complete as it needs to be.

"Although our credits database is growing it is a work in progress and is not nearly as comprehensive as it needs to be."

"Although our credits database (which is powered by our sister site GameFAQs) is growing, as our users' feedback has indicated, it is a work in progress and is not nearly as comprehensive as it needs to be to accurately provide a career score for these individuals.

"As such, we have removed that career score from the pages dedicated to creative individuals behind games on Metacritic," wrote the company on its official blog.

Many developers had complained about the relevancy, legality and methodology of the move by the site, which is regarded as a barometer of taste and is also used by publishers to negotiate bonuses with development staff.

Despite pulling the scoring system from the site, Metacritic still believes it's a feature worth pursuing and it is likely to return in the future.

"In addition to creating dedicated pages for corporate publishers and developers, on a given game's Details & Credits page, Metacritic displays those individual people who contributed to the games in our database, including designers, programmers, producers, voice actors, and artists. In turn, we have produced dedicated pages for those individuals featuring their games and associated Metascores, and, until today, their individual career scores.

"We are still very much committed to building a credits database, and welcome your participation in that process," said the site.

Related stories

Koch and Square Enix clarify review code policy

“We would never impose a blacklist or ban on any media for a review score”

By Christopher Dring

"The end of comfortable publisher-journalist relationships will lead to better journalism"

The impact of publishers going direct to consumers is having a liberating impact on the games press

By Christopher Dring

Latest comments (34)

Lee Hansiel Lim Game Developer - Unity3D, Anino PlayLab5 years ago
Well that was fast..

There was too much at stake anyways, too many variables to consider, and yet too little of them exposed to warrant 'perfectly' reliable ratings on how devs performed.

Although, I must admit, I was interested on what could've been the result(s) after a few months of letting this feature live.

Edited 1 times. Last edit by Lee Hansiel Lim on 29th March 2011 9:38am

0Sign inorRegisterto rate and reply
Phil Hindle Technical Director, FreeStyleGames5 years ago
Credit to MetaCritic for actually doing something about it promptly.
0Sign inorRegisterto rate and reply
Private Industry 5 years ago
So thats why I wasnt there, they dont do pages for QA people or other low positions. :-D

Good choice to take it down.
0Sign inorRegisterto rate and reply
Show all comments (34)
Drew Northcott Senior Artist 5 years ago
Next up - Metacritic scores for the individuals studio management.
0Sign inorRegisterto rate and reply
Fran Mulhern , Recruit3D5 years ago
Let's remember though, they didn't take it down because they've stopped believing in their "product". They took it down because they felt they didn't have enough credits to base scores on. It'll be back.
0Sign inorRegisterto rate and reply
Ted Bradley Programmer, Team 175 years ago
the logical conclusion this will come to is rating consumers based on which games they buy :p

Edited 1 times. Last edit by Ted Bradley on 29th March 2011 10:41am

0Sign inorRegisterto rate and reply
Filipe Duarte Pina Director, Nerd Monkeys5 years ago
I think a line should be drawn here because the effects of having people scored on Metacritic will have bad consequences. Even if it is accurate in regards to games scores there is no telling if it is actually correct. A very good artist might have worked on a very bad game.
0Sign inorRegisterto rate and reply
Phil Hindle Technical Director, FreeStyleGames5 years ago
@Werner - don't feel too bad about it, I wasn't on there either. Little ol' me, the lowly technical director for DJ Hero 1 & 2, and only 16 years in the games industry... :-)
0Sign inorRegisterto rate and reply
Klaus Preisinger Freelance Writing 5 years ago
Next Metacritic will score review aggregating sites. Of course they'll take it down after criticism of scientists claims rating anything on a one to ten scale is the lead cause for global warming.
0Sign inorRegisterto rate and reply
Oh dear, they've not learned a single thing, have they? No indication that they actually took the real criticisms on board: namely that while games are soulless lifeless abstract entities with a distinct purpose that you can happily take aggregate reviews on, the developers behind them are real actual people with lives and futures that often don't actually get to choose what games they work on.
0Sign inorRegisterto rate and reply
Lee Walton Co-Founder & Art Director, No More Pie5 years ago
stupid idea from them- I also didn't appear there, despite working for 10 years in this industry, not always on AAA but I have done a few. This doesn't of course take into account the lovely industry practice of removing devs from credits if they leave the company even days before a title ships (I've experienced that, although I left half-way through a project, after doing a whole years hard work on it). Plus of course, as everyone is saying- "lowly" staff members doing great assets (such as 3D prop modellers) have no input at all into the final playable quality of most games.
0Sign inorRegisterto rate and reply
Jon Gregory Studying Comp Sci, University of North Carolina5 years ago
Now if we could only stop this and the equally asinine practice of translating non-number scores into a 100 point scale. Last time I checked, a 4 star game is a 4 star game... not an 80/100 game. Metacritic is stupid, their aggregate scores are wildly inaccurate. No one should judge anything based on Metacritic unless it falls into the extreme ranges.
0Sign inorRegisterto rate and reply
Ian Williams Senior Artist 1, THQ Digital Studios UK5 years ago
Hear, hear @williamrobinson - when it comes to individuals they are in very iffy legal territory.
0Sign inorRegisterto rate and reply
John Donnelly Quality Assurance 5 years ago
How much traffic I do you think they have generated in the last 2 days?
I say their vists have spiked alot with us all going to check it out.

That is a win for them as they can use those stats with the advisiters and marketing of the site.
I stand by my comment from yesterday in that they are trying to drive more site traffic with stats like this even if we in the industry thinks it not good, the consumer who thrives on this stuff would have loved it.

I bet it drove more traffic on the sister sites and lots of comments on gamefaqs and the like on who was better than who while it was up.
This was the goal and it will make people who ignored the site before sit up and take notice.
0Sign inorRegisterto rate and reply
Sean Rogan Freelance Journalist 5 years ago
'Let's remember though, they didn't take it down because they've stopped believing in their "product". They took it down because they felt they didn't have enough credits to base scores on. It'll be back.' - Fran

I disagree. I think Metacritic are simply saving face.
0Sign inorRegisterto rate and reply
Wesley Williams Quality Assurance 5 years ago
It's pretty harsh to tarnish someone's reputation based on the opinion of the overall end product from a select number of critics. What about how many units the game sold, Just Dance comes to mind as a perfect example of a critically slated game that is loved by just about everyone who owns it. What about the camera implementation being god awful and yet the sound engineers did an awesome job, should they be punished because the guys who coded the camera were complete muppets. It was a bad idea and it will remain so because the data it needs to be based on will never be close to being accurately representative of the quality of work.
0Sign inorRegisterto rate and reply
Greg Wilcox Creator, Destroy All Fanboys! 5 years ago
Feh. I think they took it down because of the smart folk here and other industry sites, then issued a piss poor alternate reason for doing so just to save face. Why take them at their word when the majority of you were correct?
0Sign inorRegisterto rate and reply
Tim Wright Managing Director, Tantrumedia Limited5 years ago
Apparently they will also be polling sexual partners to score your love making out of 100.
0Sign inorRegisterto rate and reply
Rick Cody PBnGames-Board Member 5 years ago
I think this helps protect Metacritic's integrity. They took it down and admitted to their mistakes.
Wesley, that's why there's a readers score section. If the press don't forgive the sloppy camera the readers might.
0Sign inorRegisterto rate and reply
Andrew Goodchild Studying development, Train2Game5 years ago
If this comes back, it will be interesting if after decades of directors who disown films being substituted as "Alan Smithee" whether he gets blamed for bad games as people try to protect their score.
0Sign inorRegisterto rate and reply
Dustin Sparks Interactive Developer / Gaming Blogger 5 years ago
What's this world coming to? Next they will start rating games based on their sales out of 100 of course.
0Sign inorRegisterto rate and reply
Felix Gladitz5 years ago
I like the idea - but not for ratings sake - which throws this thing pretty much out of the window in the shape it is now.

But having a platform, an overview of some sorts - to see your connection to various projects does sound interesting to me.

Especially for people who are either just getting started in the industry or find it difficult to be heard - it's an interesting concept - when done right!

Sort of like a "Facebook" just for the Industry.

But at the same time I agree that a review site such as Metacritic might not be the right place for such a platform.

I suppose another question is: How much does a game that you've worked on, weigh in on your experience?

At what point does it all go out the window anyway? Or does it ever?

Is there a need for such a platform?

Those are my two cents really.

Edited 2 times. Last edit by Felix Gladitz on 29th March 2011 5:21pm

0Sign inorRegisterto rate and reply
Chris Charla Director ID@Xbox, Microsoft Studios5 years ago
This system is really vulnerable to griefing. Given the crowd-sourced nature of GameFAQs, it would be easy to add a name to the credits of a high-scoring game, or whatever.

I think also about a friend who paid his dues cranking out dozens of (low-scoring) licensed titles before moving on to one of the top franchises in the game industry, where he's made three games in 8 years or whatever. What's his "rating?"

The notion of just using a standard average against every game you've worked on is moronic. His 58 on a GBC game as associate producer weighs as much as his 94 on [Mega Franchise] as lead mulitplayer designer? It's not even the same discipline!

Frankly, the folks at MetaCritic should work on making sure they transcribe the review scores with 100% accuracy before they get ambitious in this area. I had a recent game get slaughtered by a rogue 40% review, which, on investigation, was actually a 4 out of 5 star review. Nicely done, Metacritic!
0Sign inorRegisterto rate and reply
Matthew Farber Senior Gameplay Engineer, Spark Unlimited5 years ago
Heh, hadn't even heard about this until now. Sounds like pretty nasty stuff. I know I wouldn't to be aggregated on some industry "hot or not" list. That just sounds all kinds of wrong.
0Sign inorRegisterto rate and reply
Iain Key Technical Director, Pi Computer Solutions Ltd5 years ago
Ouch, I missed it as well.
"Lawsuit waiting to happen" springs to mind
0Sign inorRegisterto rate and reply
Greg Wilcox Creator, Destroy All Fanboys! 5 years ago
@Matthew: especially if you're a NOT... XD

Yeah, the idea stunk to high heaven and low hell. Good riddance (and hopefully, Metacritic sends that 14-year old who thought of this to his room with no Hot Pockets).
0Sign inorRegisterto rate and reply
Kirill Yarovoy Game designer / Narrative designer / Writer 5 years ago
I wonder what score would Bobby Kotick have if players would judge him ? ^_^ ) Trollface.jpg
0Sign inorRegisterto rate and reply
Bonnie Patterson Narrative Designer, Writer 5 years ago
Using Metacritic as any sort of basis for hiring or pay bonus decisions seems to me decidedly wonky. After all, reviews take no account of budget and timely delivery, and there have been relatively many games of late that have had exquisite reviews and only sold a (relative) handful of copies.

@Tim Wright - *cough* Could you, uh, link that page when they do?
0Sign inorRegisterto rate and reply
Private Industry 5 years ago
Phil I don`t feel to bad about it, I know what games where great and I was fortunate to work on and what not in my 8 years now. I learned the most stuff from my early days with games ranking like 50/100 or below that just displays how out of tune that scoring is. Probably for most of the people it`s the same the first games where just bad, you couldn`t start at AAA titles there just wasn`t/isn`t the choice when you start. But the stuff learned on those not so amazing games while doing functional QA, 1st party requirements, content creation and a bit of level and game design really taught me a lot if not even the most and that`s vital for how I do my job now.

Really my first shipped game was a lightgun shooter for PS1 probably never sold outside the German speaking marked called Moorhuhn X, and I overlooked something like "PlayStation version" in the credits and not that good if you are the only one checking the 20 or so pages of requirements, anyway the time at that small independent developer that made games that didn`t got high scores was way more important for me than working on a good game that scores an 80 years later. No doubt I`m proud of working on games like FF12 or Blur and many more, but work experience wise I learned like 80% at the small games that scored horrible bad. Those "bad" project might have given me knowledge that people might not have who went directly into localization and help those people and the team. So yeah my worst game has probably below 50 and the best above 90, but I only got to work on that 90+ game(s) because of what I learned on those below 50 games. Not to forget my main work influences anyway only a tiny little bit the scoring of the German version if at all. I do my part of the job, but if you really want to score I don`t think I should get the same as any designers or programmers of the games. I still stick with it`s a team effort, but there are some people who have more influence than others on the final outcome and obviously I have a tiny tiny little bit of influence there so yeah if someone says the German localization in Blur was great and gives a higher score because of that yes along the translators and the other team mates that group can take credit for it. If you rate the game just by it`s gameplay I couldn`t take any credit for that and the final score does not show how good or bad any of us worked. The same goes for programmers compared artists and so on and so on. There is just no way to get a system that scores individuals running and working fair.


@Bonnie: I`m rather sure he was joking, but sarcasm does not translated to well into the internet :)

Edited 1 times. Last edit by Private on 29th March 2011 11:04pm

0Sign inorRegisterto rate and reply
Camila Chimello Writer 5 years ago
Well, i see this as a good thing at the beginning, but if needs more work and more accurate data from developers, then they act correctly removing it for now.
0Sign inorRegisterto rate and reply
John Donnelly Quality Assurance 5 years ago
Warner I know how you feel.
I have worked on some fantastic games, Crimson Skies High Road to Revenge and Geometry Wars Galaxies stand out as epic games to be credited with, yet Mach, Miami Vice and.. Dare I say it the PSP version of Scarface.. Not so much.

Sometimes is the not so good games that teach us the most, but high budget AAA's do have their places.
0Sign inorRegisterto rate and reply
Private Industry 5 years ago
Sure AAA games have their place and it`s a great experience to work on any of them, didn`t work for a really low salary to keep doing those small not so good games. The plan was simply get better gain experience and move to doing something on great games. In the end I`m from Austria with a not so good development environment with the best know company being JoWood and not for positive reasons and I got more than I hoped for. My achievement is to still being here and if I would be getting some score on how games where reviewed it would just be insulting for me and most likely for many others. It`s not some simple office job for anybody no mater how bad or good the games are and all the people put a lot of work, energy and overtime into the projects and giving them some number is just a bad joke and simply a big pile of bullshit to use some honest words as I`m usually not that outspoken publicly. I know people who do a great job and are very dedicated but didn`t work on any great games that doesn`t mean they don`t do a good job. That`s why I always feel bad for people if a game doesn`t do well and they lose the job because I had that already myself and even if the game was bad the people still put a lot of effort into it and you can`t put some number on that, the most people I know would deserve a 120/100.

Whoever came up with that stupid and idiotic idea of giving the people a score should be hired and forced to work on any kind of project from beginning to end and I want to see that person than to still say giving a score to people based on how games where reviewed is a good idea after they went trough the full process of making a game. I usually tend to not use bad language publicly but seriously that stupid idea is just getting on my nerves because on the worst games I was on people still put a lot of effort and heart into it and metacritics attitude of "not nearly as comprehensive as it needs to be to accurately provide a career score for these individuals." is just pissing me off. There is no fucking way in hell to provide an accurate career score for individuals. The time they find an accurate way hell will be frozen over and we all use portal guns to fast travel between point A and B while fighting the Nazi Zombie infestation.

This is just my personal opinion as an individual and if I violet any kind of good behavior code for commenting I`m sorry for that and will change my posting to reflect what is considered good behavior code for posting here and I actually had a good day today :)

Edited 6 times. Last edit by Private on 30th March 2011 1:17am

0Sign inorRegisterto rate and reply
jim ellis 2D/3D artist, design, illustration, concept artist, video editor 5 years ago
yeah - Im sure a lawyer didnt phone them at all... I'd love to sue their arses.
0Sign inorRegisterto rate and reply
Benjamin Seeberger Writer/Translator 5 years ago
Wow -- good thing its gone.

Having a score for a Developer (company) is a good idea, but personal scores... once you get stuck with a couple bad teams, your career would pretty much be sunk if that information were made public.

Obviously scanning Mobygames with a critical eye might give you the same result, but less obvious and scandalous.
0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.