Close
Report Comment to a Moderator Our Moderators review all comments for abusive and offensive language, and ensure comments are from Verified Users only.
Please report a comment only if you feel it requires our urgent attention.
I understand, report it. Cancel

Metacritic disables developer career scores

Tue 29 Mar 2011 7:00am GMT / 3:00am EDT / 12:00am PDT
MediaDevelopment

Site admits methodology is "not nearly as comprehensive as it needs to be" but is still committed to ratings

Metacritic has pulled controversial ratings from its site that scored individual developers out of 100 based on their contributions to games.

The site collates aggregate scores for games and reviews sites, and this latest move was in conjunction with information gathered by sister site Gamefaqs.com - but it admitted information was not as complete as it needs to be.

"Although our credits database is growing it is a work in progress and is not nearly as comprehensive as it needs to be."

"Although our credits database (which is powered by our sister site GameFAQs) is growing, as our users' feedback has indicated, it is a work in progress and is not nearly as comprehensive as it needs to be to accurately provide a career score for these individuals.

"As such, we have removed that career score from the pages dedicated to creative individuals behind games on Metacritic," wrote the company on its official blog.

Many developers had complained about the relevancy, legality and methodology of the move by the site, which is regarded as a barometer of taste and is also used by publishers to negotiate bonuses with development staff.

Despite pulling the scoring system from the site, Metacritic still believes it's a feature worth pursuing and it is likely to return in the future.

"In addition to creating dedicated pages for corporate publishers and developers, on a given game's Details & Credits page, Metacritic displays those individual people who contributed to the games in our database, including designers, programmers, producers, voice actors, and artists. In turn, we have produced dedicated pages for those individuals featuring their games and associated Metascores, and, until today, their individual career scores.

"We are still very much committed to building a credits database, and welcome your participation in that process," said the site.

34 Comments

Lee Hansiel Lim Game Developer - Unity3D, Anino PlayLab

26 2 0.1
Well that was fast..

There was too much at stake anyways, too many variables to consider, and yet too little of them exposed to warrant 'perfectly' reliable ratings on how devs performed.

Although, I must admit, I was interested on what could've been the result(s) after a few months of letting this feature live.

Edited 1 times. Last edit by Lee Hansiel Lim on 29th March 2011 9:38am

Posted:3 years ago

#1

Phil Hindle Technical Director, FreeStyleGames

19 37 1.9
Credit to MetaCritic for actually doing something about it promptly.

Posted:3 years ago

#2

Private Industry

1,176 182 0.2
So thats why I wasnt there, they dont do pages for QA people or other low positions. :-D

Good choice to take it down.

Posted:3 years ago

#3

Drew Northcott Senior Artist

8 2 0.3
Next up - Metacritic scores for the individuals studio management.

Posted:3 years ago

#4

Fran Mulhern , Recruit3D

863 707 0.8
Let's remember though, they didn't take it down because they've stopped believing in their "product". They took it down because they felt they didn't have enough credits to base scores on. It'll be back.

Posted:3 years ago

#5

Ted Bradley Programmer, Team 17

1 0 0.0
the logical conclusion this will come to is rating consumers based on which games they buy :p

Edited 1 times. Last edit by Ted Bradley on 29th March 2011 10:41am

Posted:3 years ago

#6
I think a line should be drawn here because the effects of having people scored on Metacritic will have bad consequences. Even if it is accurate in regards to games scores there is no telling if it is actually correct. A very good artist might have worked on a very bad game.

Posted:3 years ago

#7

Phil Hindle Technical Director, FreeStyleGames

19 37 1.9
@Werner - don't feel too bad about it, I wasn't on there either. Little ol' me, the lowly technical director for DJ Hero 1 & 2, and only 16 years in the games industry... :-)

Posted:3 years ago

#8

Klaus Preisinger Freelance Writing

1,137 1,174 1.0
Next Metacritic will score review aggregating sites. Of course they'll take it down after criticism of scientists claims rating anything on a one to ten scale is the lead cause for global warming.

Posted:3 years ago

#9
Oh dear, they've not learned a single thing, have they? No indication that they actually took the real criticisms on board: namely that while games are soulless lifeless abstract entities with a distinct purpose that you can happily take aggregate reviews on, the developers behind them are real actual people with lives and futures that often don't actually get to choose what games they work on.

Posted:3 years ago

#10

Lee Walton Co-Founder & Art Director, No More Pie

35 7 0.2
stupid idea from them- I also didn't appear there, despite working for 10 years in this industry, not always on AAA but I have done a few. This doesn't of course take into account the lovely industry practice of removing devs from credits if they leave the company even days before a title ships (I've experienced that, although I left half-way through a project, after doing a whole years hard work on it). Plus of course, as everyone is saying- "lowly" staff members doing great assets (such as 3D prop modellers) have no input at all into the final playable quality of most games.

Posted:3 years ago

#11

Jon Gregory Studying Comp Sci, University of North Carolina

4 0 0.0
Now if we could only stop this and the equally asinine practice of translating non-number scores into a 100 point scale. Last time I checked, a 4 star game is a 4 star game... not an 80/100 game. Metacritic is stupid, their aggregate scores are wildly inaccurate. No one should judge anything based on Metacritic unless it falls into the extreme ranges.

Posted:3 years ago

#12

Ian Williams Senior Artist 1, THQ Digital Studios UK

3 0 0.0
Hear, hear @williamrobinson - when it comes to individuals they are in very iffy legal territory.

Posted:3 years ago

#13

John Donnelly Quality Assurance

313 38 0.1
How much traffic I do you think they have generated in the last 2 days?
I say their vists have spiked alot with us all going to check it out.

That is a win for them as they can use those stats with the advisiters and marketing of the site.
I stand by my comment from yesterday in that they are trying to drive more site traffic with stats like this even if we in the industry thinks it not good, the consumer who thrives on this stuff would have loved it.

I bet it drove more traffic on the sister sites and lots of comments on gamefaqs and the like on who was better than who while it was up.
This was the goal and it will make people who ignored the site before sit up and take notice.

Posted:3 years ago

#14

Sean Rogan Freelance Journalist

7 0 0.0
'Let's remember though, they didn't take it down because they've stopped believing in their "product". They took it down because they felt they didn't have enough credits to base scores on. It'll be back.' - Fran

I disagree. I think Metacritic are simply saving face.

Posted:3 years ago

#15

Wesley Williams Quality Assurance

133 72 0.5
It's pretty harsh to tarnish someone's reputation based on the opinion of the overall end product from a select number of critics. What about how many units the game sold, Just Dance comes to mind as a perfect example of a critically slated game that is loved by just about everyone who owns it. What about the camera implementation being god awful and yet the sound engineers did an awesome job, should they be punished because the guys who coded the camera were complete muppets. It was a bad idea and it will remain so because the data it needs to be based on will never be close to being accurately representative of the quality of work.

Posted:3 years ago

#16

Greg Wilcox Creator, Destroy All Fanboys!

2,193 1,170 0.5
Feh. I think they took it down because of the smart folk here and other industry sites, then issued a piss poor alternate reason for doing so just to save face. Why take them at their word when the majority of you were correct?

Posted:3 years ago

#17

Tim Wright Managing Director, Tantrumedia Limited

29 0 0.0
Apparently they will also be polling sexual partners to score your love making out of 100.

Posted:3 years ago

#18

Rick Cody PBnGames-Board Member

144 14 0.1
I think this helps protect Metacritic's integrity. They took it down and admitted to their mistakes.
Wesley, that's why there's a readers score section. If the press don't forgive the sloppy camera the readers might.

Posted:3 years ago

#19

Andrew Goodchild Studying development, Train2Game

1,254 421 0.3
If this comes back, it will be interesting if after decades of directors who disown films being substituted as "Alan Smithee" whether he gets blamed for bad games as people try to protect their score.

Posted:3 years ago

#20

Dustin Sparks Interactive Developer / Gaming Blogger

13 0 0.0
What's this world coming to? Next they will start rating games based on their sales out of 100 of course.

Posted:3 years ago

#21

Felix Gladitz

12 0 0.0
I like the idea - but not for ratings sake - which throws this thing pretty much out of the window in the shape it is now.

But having a platform, an overview of some sorts - to see your connection to various projects does sound interesting to me.

Especially for people who are either just getting started in the industry or find it difficult to be heard - it's an interesting concept - when done right!

Sort of like a "Facebook" just for the Industry.

But at the same time I agree that a review site such as Metacritic might not be the right place for such a platform.

I suppose another question is: How much does a game that you've worked on, weigh in on your experience?

At what point does it all go out the window anyway? Or does it ever?

Is there a need for such a platform?

Those are my two cents really.

Edited 2 times. Last edit by Felix Gladitz on 29th March 2011 5:21pm

Posted:3 years ago

#22

Chris Charla Director ID@Xbox, Microsoft Studios

5 12 2.4
This system is really vulnerable to griefing. Given the crowd-sourced nature of GameFAQs, it would be easy to add a name to the credits of a high-scoring game, or whatever.

I think also about a friend who paid his dues cranking out dozens of (low-scoring) licensed titles before moving on to one of the top franchises in the game industry, where he's made three games in 8 years or whatever. What's his "rating?"

The notion of just using a standard average against every game you've worked on is moronic. His 58 on a GBC game as associate producer weighs as much as his 94 on [Mega Franchise] as lead mulitplayer designer? It's not even the same discipline!

Frankly, the folks at MetaCritic should work on making sure they transcribe the review scores with 100% accuracy before they get ambitious in this area. I had a recent game get slaughtered by a rogue 40% review, which, on investigation, was actually a 4 out of 5 star review. Nicely done, Metacritic!

Posted:3 years ago

#23

Matthew Farber Senior Gameplay Engineer, Spark Unlimited

8 0 0.0
Heh, hadn't even heard about this until now. Sounds like pretty nasty stuff. I know I wouldn't to be aggregated on some industry "hot or not" list. That just sounds all kinds of wrong.

Posted:3 years ago

#24

Iain Key Technical Director, Pi Computer Solutions Ltd

5 0 0.0
Ouch, I missed it as well.
"Lawsuit waiting to happen" springs to mind

Posted:3 years ago

#25

Greg Wilcox Creator, Destroy All Fanboys!

2,193 1,170 0.5
@Matthew: especially if you're a NOT... XD

Yeah, the idea stunk to high heaven and low hell. Good riddance (and hopefully, Metacritic sends that 14-year old who thought of this to his room with no Hot Pockets).

Posted:3 years ago

#26

Kirill Yarovoy Game designer / Narrative designer / Writer

41 5 0.1
I wonder what score would Bobby Kotick have if players would judge him ? ^_^ ) Trollface.jpg

Posted:3 years ago

#27

Bonnie Patterson Freelance Narrative Designer

160 437 2.7
Using Metacritic as any sort of basis for hiring or pay bonus decisions seems to me decidedly wonky. After all, reviews take no account of budget and timely delivery, and there have been relatively many games of late that have had exquisite reviews and only sold a (relative) handful of copies.

@Tim Wright - *cough* Could you, uh, link that page when they do?

Posted:3 years ago

#28

Private Industry

1,176 182 0.2
Phil I don`t feel to bad about it, I know what games where great and I was fortunate to work on and what not in my 8 years now. I learned the most stuff from my early days with games ranking like 50/100 or below that just displays how out of tune that scoring is. Probably for most of the people it`s the same the first games where just bad, you couldn`t start at AAA titles there just wasn`t/isn`t the choice when you start. But the stuff learned on those not so amazing games while doing functional QA, 1st party requirements, content creation and a bit of level and game design really taught me a lot if not even the most and that`s vital for how I do my job now.

Really my first shipped game was a lightgun shooter for PS1 probably never sold outside the German speaking marked called Moorhuhn X, and I overlooked something like "PlayStation version" in the credits and not that good if you are the only one checking the 20 or so pages of requirements, anyway the time at that small independent developer that made games that didn`t got high scores was way more important for me than working on a good game that scores an 80 years later. No doubt I`m proud of working on games like FF12 or Blur and many more, but work experience wise I learned like 80% at the small games that scored horrible bad. Those "bad" project might have given me knowledge that people might not have who went directly into localization and help those people and the team. So yeah my worst game has probably below 50 and the best above 90, but I only got to work on that 90+ game(s) because of what I learned on those below 50 games. Not to forget my main work influences anyway only a tiny little bit the scoring of the German version if at all. I do my part of the job, but if you really want to score I don`t think I should get the same as any designers or programmers of the games. I still stick with it`s a team effort, but there are some people who have more influence than others on the final outcome and obviously I have a tiny tiny little bit of influence there so yeah if someone says the German localization in Blur was great and gives a higher score because of that yes along the translators and the other team mates that group can take credit for it. If you rate the game just by it`s gameplay I couldn`t take any credit for that and the final score does not show how good or bad any of us worked. The same goes for programmers compared artists and so on and so on. There is just no way to get a system that scores individuals running and working fair.


@Bonnie: I`m rather sure he was joking, but sarcasm does not translated to well into the internet :)

Edited 1 times. Last edit by Private on 29th March 2011 11:04pm

Posted:3 years ago

#29

Camila Chimello Writer

2 0 0.0
Well, i see this as a good thing at the beginning, but if needs more work and more accurate data from developers, then they act correctly removing it for now.

Posted:3 years ago

#30

John Donnelly Quality Assurance

313 38 0.1
Warner I know how you feel.
I have worked on some fantastic games, Crimson Skies High Road to Revenge and Geometry Wars Galaxies stand out as epic games to be credited with, yet Mach, Miami Vice and.. Dare I say it the PSP version of Scarface.. Not so much.

Sometimes is the not so good games that teach us the most, but high budget AAA's do have their places.

Posted:3 years ago

#31

Private Industry

1,176 182 0.2
Sure AAA games have their place and it`s a great experience to work on any of them, didn`t work for a really low salary to keep doing those small not so good games. The plan was simply get better gain experience and move to doing something on great games. In the end I`m from Austria with a not so good development environment with the best know company being JoWood and not for positive reasons and I got more than I hoped for. My achievement is to still being here and if I would be getting some score on how games where reviewed it would just be insulting for me and most likely for many others. It`s not some simple office job for anybody no mater how bad or good the games are and all the people put a lot of work, energy and overtime into the projects and giving them some number is just a bad joke and simply a big pile of bullshit to use some honest words as I`m usually not that outspoken publicly. I know people who do a great job and are very dedicated but didn`t work on any great games that doesn`t mean they don`t do a good job. That`s why I always feel bad for people if a game doesn`t do well and they lose the job because I had that already myself and even if the game was bad the people still put a lot of effort into it and you can`t put some number on that, the most people I know would deserve a 120/100.

Whoever came up with that stupid and idiotic idea of giving the people a score should be hired and forced to work on any kind of project from beginning to end and I want to see that person than to still say giving a score to people based on how games where reviewed is a good idea after they went trough the full process of making a game. I usually tend to not use bad language publicly but seriously that stupid idea is just getting on my nerves because on the worst games I was on people still put a lot of effort and heart into it and metacritics attitude of "not nearly as comprehensive as it needs to be to accurately provide a career score for these individuals." is just pissing me off. There is no fucking way in hell to provide an accurate career score for individuals. The time they find an accurate way hell will be frozen over and we all use portal guns to fast travel between point A and B while fighting the Nazi Zombie infestation.

This is just my personal opinion as an individual and if I violet any kind of good behavior code for commenting I`m sorry for that and will change my posting to reflect what is considered good behavior code for posting here and I actually had a good day today :)

Edited 6 times. Last edit by Private on 30th March 2011 1:17am

Posted:3 years ago

#32

jim ellis 2D/3D artist, design, illustration, concept artist, video editor

27 1 0.0
yeah - Im sure a lawyer didnt phone them at all... I'd love to sue their arses.

Posted:3 years ago

#33

Benjamin Seeberger Writer/Translator

28 18 0.6
Wow -- good thing its gone.

Having a score for a Developer (company) is a good idea, but personal scores... once you get stuck with a couple bad teams, your career would pretty much be sunk if that information were made public.

Obviously scanning Mobygames with a critical eye might give you the same result, but less obvious and scandalous.

Posted:3 years ago

#34

Login or register to post

Take part in the GamesIndustry community

Register now