Close
Are you sure? Are you sure you want to report this comment? I understand, report it. Cancel

Defining Success: Why Metacritic Should Be Irrelevant

Defining Success: Why Metacritic Should Be Irrelevant

Mon 13 May 2013 2:20pm GMT / 10:20am EDT / 7:20am PDT
Development

Warren Spector tackles the thorny issue of Metacritic's impact on the industry in his second exclusive column for GamesIndustry International

To qualify for column treatment here, a subject has to involve something I've been thinking about a lot - a problem I can't solve or a question I can't answer. This time, I started with a question that's plagued me for years (yes... even before the reviews started coming in on Disney Epic Mickey!). That question is "Why does everyone care so much about Metacritic?"

But as I started thinking about turning that question into a column actually worth reading - something that wouldn't sound like sour grapes or clichéd developer whining - I realized that the thing I was really thinking about, the thing we should probably all be thinking about, was bigger than that. The topic worth talking about was how we define success in gaming, how we measure it and how we use that measurement, for good and ill.

I started thinking about ratings and rankings, yes, but also about how reviews relate (or don't) to critical analysis and about how different parts of the game business balance art and commerce. Clearly, Metacritic is part of the answer, but it's just a part. I'll still talk about Metacritic, some, but it won't be the only thing I talk about, as I originally intended.

So, let's talk about success.

Obviously, there's personal, individual success. You know, things like Fame (people recognize you), Respect or Significance (people acknowledge your worth and recognize your contributions to your field), Wealth (people see you driving a fancy car or - ahem! - owning several houses)... These are interesting but a topic for a therapist not a game developer like me. That's not the kind of success I want to talk about here.

What I am interested in is how we define an individual game's success. With thousands of games produced and offered to players each year, this is a vital question - for developers, for publishers, for players and even for critics and historians whose job is to explain our medium to us and to future generations. How do we balance and describe commercial success vis a vis critical success and what might be called significance? How do games relate to other media? And how do we communicate how a game fits into - or stands out from - the continuum of games and media?

Needless to say, these are huge questions and that hugeness allows me to rationalize the fact that I'm not close to answering any one of them, definitively. That, of course, is where all of you come in: I'll lay out my thoughts here, hazy though they may be, and then you get to tell me where I've gone wrong. Don't let me down!

Games as Commercial Art

Clearly, games are a commercial art, and success in a commercial medium requires that we focus on the "commercial" side of the equation. A game has to generate some amount of revenue and, more important, profit.

But what about the "art" side? My personal approach to game design is to offer players a variety of ways to act and then, as much as possible, allow them to define success for themselves. Why should the various stakeholders in the creation of a game not have similar power? Well, clearly, they shouldn't. The term "commercial art" is nicely ambiguous, capturing the idea that individual stakeholders might have very different definitions of "success."

1

Epic Mickey 2

For example, most if not all publishers think first about revenue and profit - no surprise, it's their job to do so. That's a relatively simple measure of success, right? Did the game make money - yes/no? (In reality, the determination of profitability is surprisingly difficult; nothing's ever simple where an Excel spreadsheet is involved, but conceptually, you get what I'm saying...)

Players have a different definition, typically putting a variety of poorly defined characteristics at the top of their lists - "fun" (whatever the hell that means), "playtime" (duration of play and value for money), peer approval (we're all playing the same games and can experience and talk about similar experiences).

A lot of developers fall in line with publishers and put revenue and profit at the top of their definitional lists as well. Others put creativity or personal expression up there.

My point is, when you go into any project, you need to know what you're shooting for. Someone gets to make that call. If it's you, your financial backers need to know what you're shooting for. If it's the biz guys, you need to know what return on investment they're expecting. Regardless, the development team needs to know. This is a minefield that must be traversed clearly and carefully. The arc of your career (even whether you have a career) hinges on understanding this and negotiating with stakeholders until you're on the same page.

I've been lucky - or foolish enough - that I've been able to define success largely for myself. And my definition rarely begins with questions of commercial appeal, sales, revenue or profit. (Okay, to be honest, I've never started there...) Before a development deal closes I tell every money person and publisher I work with (or even think about working with) one thing:

My obligation is to sell one more copy of a game to get someone to fund my next one. (Does saying this publicly ensure that I'll never work again? Why do I do this sort of thing?)

I'm not saying that commercial considerations are beneath me. You have to make some money or it really is "career-over." But if you look at the mission statement I've used at Ion Storm and Junction Point you'll see that some things are higher priorities than the unholy trinity of Sales, Revenue and Profit.

"The fact is, few of us will ever approach that Platonic ideal of art, commerce and critical acclaim achieved in equal measures"

For example, the projects I've chosen to work on put "player expression" at the top of the list - if players aren't telling their own stories, creating unique experiences through their playstyle choices, the game just doesn't seem worth making. It doesn't matter how much money a different kind of game might make. My success definition prioritizes something above that.

Similarly critical to me is "innovation," defined as including something in the game no one has ever seen before. To my mind, it's better to fail at something new (and worth doing) than it is to succeed by executing well against a well-understood problem.

Starting a game without trying for the highest level of quality seems kind of soul crushing, though achieving what might be called "Game of the Year quality" is rare and wondrous. And great review scores are always nice. But these come somewhere after player expression and innovation, and ahead of money. For me. Your mileage may vary.

At the end of the day, we all want our work to "succeed" in every way, at all levels, with all people. We all want to create art that sells like hotcakes, makes a ton of money and wins the admiration of consumers and critics alike. Anyone who says "I want to make games that reach the smallest possible audience" is probably lying, and anyone who says "Screw making art - I just want to make a ton of money" probably has no chance of making either.

The fact is, few of us will ever approach that Platonic ideal of art, commerce and critical acclaim achieved in equal measures. And that means each of us must decide for ourselves how to define success. For me, that means achieving a kind of self-satisfaction - knowing my team accomplished the goals we set as effectively as we could, given circumstances. For me, that means living up to the priorities captured in a personal mission statement. For me, that means doing well enough, commercially, that someone will step up and fund my next game.

But over the years, review scores - even pre-release score predictions and especially post-release aggregate review scores - are, more and more, accepted as the most significant measure of success. Publishers use them to determine marketing and PR budgets, print runs and distribution plans and even developer royalties.

Pre-release, consultants are often hired these days by publishers who feel the need of impartial evaluations of games in development. Such consultants offer assessments of gameplay, story and graphics, as well as a prediction of post-release review scores. These services are typically offered by ex-reviewers whose individual opinions carried far less weight when they were writing than they do after they stop - something I'll go to my grave not understanding. To be blunt, in my experience, these consultants rarely reveal anything about gameplay, story or graphics that developers don't already know.

But publishers, who should probably trust themselves and their experience a bit more, do give great weight to these evaluations, assuming they actually reflect what will be post-release review scores. But, more important, they use these assessments to predict a finished game's likely level of commercial success.

Why does this matter? Well, as I said, such predictions drive marketing budgets and PR plans as well as expenditures that determine shelf space in stores and so on. In other words, review score predictions become self-fulfilling prophecies, with games performing as expected as a result of the use of predictions to drive plans.

After release, the assessment of a game's success passes out of consultants' hands - even out of the hands of internal testing and evaluation resources - and into the hands of two groups with different but related agendas:

First, assessment of a game's "quality" falls into the hands of gamers who, pleased or displeased enough, may express their opinions in online forums and in conversation with friends.

In addition, of course, a game's quality is assessed by game reviewers, whose opinions appear in consumer and specialist print magazines, mainstream newspapers, and on websites and television programs.

Finally, reviews are aggregated into a single "score," determined by sites like Metacritic.

Obviously, gamers have every right to evaluate games themselves, and to share their opinions with other gamers. It would be insane to argue against that, or even to think about it too much.

Reviews and reviewers? That's a very different story. We - gamers, developers, publishers and reviewers themselves - need to think more about the purpose of reviews. Only by doing this will we get better reviews and understand how best to interpret them, as business people, creators and consumers.

In gaming, there's a widely held, if largely unspoken belief that the function of a review is to tell players whether a game is "good" or "bad." There's usually enough description of genre, story and/or gameplay to inform players what kind of game they're dealing with. Reviewers typically offer reasons for liking or not liking a game.

But, fundamentally, game reviewers exist, it seems, to say whether a game is good or bad, with little concern for the success criteria I went on about earlier. The end result is a score of some kind - 96 out of 100... 8 out of 10... a B+... 3.5 stars... A game is good or bad.

Frankly, that's not good enough and it misses the point of reviews, to my way of thinking.

"When we put our faith in Metacritic as an impartial, scientific measure of quality, we should probably ask ourselves whether the crowd - the crowd of journalists as well as players - is really wise or just mediocre, incapable of recognizing and rewarding the new and different"

When I read a review by pioneering film critics (like Roger Ebert, Pauline Kael, Judith Christ, etc.) or any of a number of more modern film reviewers I'm, of course, interested in their good/bad assessments. After all, these are smart people whose expertise gives them a status that makes their opinions worthwhile. However, the real value of their reviews is something very different, something more valuable - something I'd argue is largely missing from game reviews today.

That valuable thing is a consistent critical voice. Great critics don't focus on questions of good or bad. Their assessments of a film's success or failure are supported by an underlying philosophy they apply to all films. It doesn't matter if you agree or disagree with their assessment of an individual film - heck, they often disagreed with one another. What matters is that, reading them, you can weigh your own likes and dislikes to determine if you will like a film, based on their review.

Frankly, the best film reviewer for me wrote for the Austin American-Statesman many years ago. He was a bad writer and I disagreed with his reviews pretty much all the time. But that was the point - I could tell reading his reviews whether I was going to like a film or not. He had a consistent voice. He loved a kind of film I knew I didn't like and hated a kind of film I loved. He didn't tell me if a film was good or bad - he told me whether a movie was right for me, something a score or numerical rating wouldn't have done. That was a hugely valuable service.

To provide that service, a critic needs a consistent voice, a philosophy applied to all games - and these characteristics need to be clear to readers. And I see all of this as being lacking in today's game review world. An enthusiastic "This game sucks!" or "The AI is bad" or "This game gets a 4 out of 5" tells me next to nothing I need to know.

So what does any of this have to do with Metacritic, the thing that got me started thinking about this in the first place?

There's no questioning the appeal of aggregating lots of individual reviews to get at some normalized measure of that subjective thing we call "quality." In today's era of Big Data, many people feel that everything can be measured, quantified and scientifically studied. To an extent, they're right.

As one developer (who asked not to be named) put it, Metacritic "is a fine cross section of established voices and existing trends, but it often misses things that reach other audiences or try new things. Too many of the sources only reward things they feel safe rewarding... We continue to believe there is a correct mono cultural taste. Bad for discoverability and bad for diversity."

By this logic, Metacritic, at best, rewards games that are conventional and well understood by players and critics alike. New and challenging things are, by their very nature, disruptive and easily misunderstood. Aggregation of opinion, at best, offers hope and guidance to people whose goal is to maximize profitability but little to people whose priorities lie elsewhere (see earlier discussion of definitions of success) or who depend on the constancy and relative predictability of the status quo.

When we put our faith in Metacritic as an impartial, scientific measure of quality, we should probably ask ourselves whether the crowd - the crowd of journalists as well as players - is really wise or just mediocre, incapable of recognizing and rewarding the new and different. (Before you come back at me with arguments, look at the highest rated games and then we'll talk...)

Beyond questions of the utility of its data, there are two aspects of Metacritic's methods that undercut its credibility in my eyes and, should, I think call its accuracy and even the validity of its data into question.

First, the aggregation of data is skewed by the selection of review sites included in the aggregation. For example, with Disney Epic Mickey, I know of several perfect scores (higher even than I would have given the game!) that were simply not listed or included in the game's average. And we're talking about high profile, well-respected sites here. By way of contrast, our worst review scores, typically from Some Guy With a Website, were integrated in the aggregation instantly. I know this sounds like sour grapes and if you want to interpret this in that way, you're welcome to do so. But with future projects, team bonuses, and so on hanging in the balance, I don't think it's inappropriate to ask for a public discussion of the reasoning behind reviews that are included in Metacritic's calculations and those that are not.

Second, the data are skewed by the weighting and summary-conversion systems employed by the people behind Metacritic's ratings. Though I have no idea what criteria are applied or the decisions made, my understanding is that certain reviews are given greater weight in determining a game's overall score. I applaud the concept here, but without knowing exactly how the weighting is applied, the validity of the score is called into question.

Similarly, the widely discussed conversion of certain reviewers' letter grades into a numerical rating is arbitrary. And the conversion of 1-10 scales and 1-5 star ratings to Metacritic's 1-100 point scale introduces a host of problems. Is a B and 80? Who decides? Is a 3 star review a 60? Again, who says? And what are the ramifications of making that decision?

Now, I realize that all of these questions about data are, to an extent, evened out when one considers that the procedural limitations apply to all games equally (at least one hopes so...). But then we're back to the question of whether the aggregation of several review scores tells us anything of value. What exactly is it measuring and does this measurement reflect one, several or none of the success criteria we can apply to a game?

Again, using my own games as examples, Metacritic provides some interesting insights.

The highest rated games I've worked on are Deus Ex and the Thief games. That's great. I'm hugely proud of the teams that made those games great (at least in my eyes!). Who doesn't like great scores, the awards that flow from them and the credibility with publishers they provide?

2

Thief: The Dark Project

By contrast, the Disney Epic Mickey games are the lowest rated games I've worked on. I'm still proud of the teams and of the games themselves.

And you know what? Despite significantly lower Metacritic scores, Disney Epic Mickey is the best-selling game I've ever worked on, by a substantial margin. And the sequel, Disney Epic Mickey: The Power of Two, is the second best selling game in a thirty-year career.

Reviewers and some gamers may have preferred Deus Ex and Thief (and I have the fan mail to prove it!), but I received substantially more fan mail - and more heartfelt messages of thanks... and more fan art... and more everything - on the DEM games than on both DX games I worked on combined.

Metacritic be damned - I'll take an emotional connection with players and the praise of Disney fans any day of the week. My definition of success - notably empowering players to craft unique experiences and offering players things they've never seen or done in a game before - were well served by games that unarguably Metacritic'ed poorly.

So where does all of this leave us? I guess I'd say this:

Make sure you define success on each project you undertake as surely and precisely as you define your gameplay, technical, artistic and, yes, fiscal constraints. That definition of success will guide all your development decisions at least as much as your elevator pitch high-concept will.

If you're lucky and/or stubborn, define success for yourself and pay no heed to the desires of outside parties or the tyranny of the aggregated audience.

If you're not lucky (I assume stubbornness will be no problem for game developers!) you'll have to include collaborators in your definitional efforts, but don't let go of your agenda, even if you have to keep it hidden. Personal satisfaction depends on personal goals, and game development is too grindingly difficult to sacrifice personal satisfaction.

Whether personally or collaboratively determined, make sure everyone involved in the project understands the success criteria so results can be measured. And then make sure the appropriate measurement tools are being used. If that happens to be Metacritic, or some other review aggregation service, know that's the case going in. If the right tool is a close reading of fan mail to track player impact or emotional engagement, know that. If your goal is to influence other developers, know that, and find appropriate measures of success (citations by other developers, the opinions of trusted critical voices, etc.).

At the end of the day, understand that "success" is a word with as many meanings as there are people to define it. Choose your meaning carefully and live with the joy - and consequences - of your choice. In this way, life is like a game. If only more games were like life. Accomplishing that goal, making games more like life and encouraging others to do the same, is my definition of success - what's yours?

Warren Spector left academia in 1983 and has been making games, as well as lecturing and writing about them, ever since. You can follow him on Twitter under the user name @Warren_Spector.

31 Comments

I agree Metacritic is not a perfect tool. And I strongly agree that publishers and pundits read too much into it.

However, in answer to your question: "Why does everyone care so much about Metacritic?" let me express the impulse behind that caring.

What folks who make and sell games are desperate for is some external, objective assessment of how others (those with no vested interest) view their game. That desire might be ill-served by how Metacritic actually works, but I understand completely the wish to use Metacritic or Game Rankings or something similar.

Warren, I think you know that, but brush it off as being driven by a profit motive that has wrongly replaced a quality/expression motive. But that is not always the case. For many of us, we really are looking for a quality assessment, period.

What I tell my teams is that the only thing you can really control is the quality of what you create. The market is a queer entity that imperfectly rewards quality. So while from a business perspective, we need our games to sell, if we pervert our creative efforts to directly chase sales, we are very likely to blow it. This is the fate of most licensed games and it is a real tribute to your work, Warren, that your teams avoided those pitfalls.

But if all you can control is the quality of what you produce, then you will want to define what that is and measure it. And one measure is critical reception. Sites that aggregate critical reception therefore provide a tool and service that is (at least theoretically) useful.

Here's the flip side of completely ignoring Metacritic, Game Rankings, reviewers in general... Ego mania, self absorption and arrogance. And the reality is that those confident enough to be creators will be prone to these issues. Indeed, a certain amount of ego must be cultivated to be a creator. But we must be wary of our inherent tendency to believe in our vision despite feedback that suggests we are on the wrong track. A balance has to be found. I believe this has been issue for our industry throughout its history. As one example, for how many decades did our industry make games for ourselves and ignore the casual gamer?

I think your specific issues with Metacritic are completely valid. But ignoring the reality that we are always looking for (and need) objective feedback isn't the answer either.

Posted:A year ago

#1

Lewis Pulsipher
Game Designer, Author, Teacher

25 33 1.3
Any review, whether of movies, games, books, software, or magazines, ought to answer three questions:

1) What is the author/creator trying to accomplish?
2) How well did he or she do it?
3) Was it worth doing?

But the reviewer must always explain WHY he or she answers these questions a certain way. Otherwise the review doesn't achieve the purpose, which is to help the reader decide whether to partake of the reviewed product.

My main exposure to video game reviews is through PC Gamer and GameInformer. It's fascinating how often the reviewer assumes that his tastes match the taste of his readers, as though there was only one way to enjoy games. So they don't bother to say "this wasn't fun for me", they just say "this wasn't fun". I sometimes wonder if the reviews are so short that the reviewers don't feel they can recognize differences in taste.

Numbers can only reflect the tastes of the reviewer.

Posted:A year ago

#2

Lewis Pulsipher
Game Designer, Author, Teacher

25 33 1.3
As for why Metacritic is relied on so much: in the USA, at least, we've been conned into thinking that only quantitative results can be accurate or reliable. A game review by its very nature ought to be qualitative, but it's hard to aggregate or even keep track of qualitative evaluations, hence the reliance on the quantitative. (This is why multiple choice tests are so popular in the American education system: yet "life is an essay test, not multiple choice".

Posted:A year ago

#3
Great article Warren.

In medicine, we use meta analysis to guide our existing best policy treatment, and in games, this loosely can be useful as a tool via aggregated analysis of meta critic. However, its somewhat misguided for studios to go out and skew games to produce a good meta critic review, prior to a game launch. In addition, the FINAL product (which may have favourable metacritic scores) may not even play or mesh well altogether compared to the demo version, which does not provide a real safe landing zone for producers/ and the teams involved. It might not even translate into a commercial success.

So, at the end of the day, sure be guided somewhat by the aggregated score, but ultimately its better say Metacritic be damned & produce the most polished game you can with each iteration. And pray, it sells a copy or so somewhere. Who knows, it might be a sleeper summer hit someday, somehow - because I dont think there is a way to predict or guarantee success.

Posted:A year ago

#4

Patrick Block
gaming conceptual artist / comic artist

2 6 3.0
Popular Comment
In my opinion, the nature of lumping together reviews destroys the entire point of the exercise of personally reviewing a game, movie, book, or other piece of art.

Art ought to offer the recipient some unique vision of human existence. Sure, games need to be fun- entertainment is foremost on people's minds- but, the memorable piece of art is almost always founded upon how it is different from any piece of art you have encountered before. There is a subtlety that is hard to quantify that people address in a good review, a sense of wonder, of awe at this construction that dazzles you with it's unique approach. How often does this sort of description appear when you slop all the reviews together and dumb down a game's appeal to single syllable words for common consumption?

This tendency to use broad comparisons and hanging onto conventions and genres well established in the gaming world when reviewing just narrows people's focus on what a game can accomplish. It adds up to a reliance on familiarity and the repeating of what is familiar and easily identifiable. It leads to franchises, series, and formulae,

I think that the more formalized and united the reviewing process is, the smaller and smaller the publisher core becomes. Eventually it leads to two or three big game franchises and spin-offs, and nothing else.

Creativity does not sit with the publishers and suits. They simply try and follow the dollars. If change, aesthetics, real creativity is going to be invested in new games, it's up to the the game designers and creators, and nowhere else. We need to concentrate as much as we can on offering things that have not been done before in quite the same way, despite publishers and reviewers apparently wanting more of the same. It's certain that the "creative spirit" resides with us, and nowhere else. This is why kickstarted games by creators themselves, funded by the fanbase, seems so appealing to me.

Art has always been plagued by these same problems. When the classic artisians like Michelangelo and Da Vinci worked, they faced the same damn thing, People trying to get as much work out of the guys as they could, trying to make as much money off them as possible without regard to the art. It wasn't the artist's patrons that inspired greatness. These artists could have settled for less, and made extra cash in doing so. It's up to the artist to fight against the suits who don't actually care or even know what art is.

Warren's approach is dead on. A creator needs to have the confidence in his own taste and artistic skill to commit to what he believes in, and follow his vision. Sure, there has to be a bit of grounding or maturity in understanding what can be accomplished, (as in published), but beyond this, no one but we creators are going to create.

And, I believe, astute fans will see the truth. Some consumers care more than we might think. I am often surprised when someone comes along and I find out that they know more about my creations than I do, having forgot about this idea or little nuance in my work ages ago. That feeling of....accomplishment in bringing to someone else you never met...a moment or two of bliss and his or her appreciation of your efforts is worth more than money.

So, while we sometimes fail in our commercial skills at raising millions, I can appreciate Warren's note that there is more to success than a dollar, and in some ways, that harder to describe success is the most important sort.

If you brought Da Vinci back to life, and asked him his regrets, would he complain about bowing to the almighty lira, or at putting too much attention to each creation?

Posted:A year ago

#5

Gregory Hommel
writer

91 53 0.6
I think most gamers and developers know that Metacritic is the wrong way to define a game. The relationship between publishers and reviewers is suspect to say the least. It is an unacceptable conflict of interest that needs to be addressed.

The problem is that those with the power to spread the word are often the publications and websites that contain the reviews. I can't believe we have entered an age where contracts include stipulations regarding aggregate scores.

Most gamers would agree that top sellers such as the Call of Duty franchise and Assassin's Creed have gone stale as they have milked their respective engines over and again. Though their aggregate scores don't reflect that sentiment in the least.

I hope light is shed on this issue because it hurts the best and brightest developers and rewards those who seek only to separate consumers from their money. The industry as a whole has already suffered from it and it only gets worse from here.

Thanks to Mr. Spector for his unabashed assessment of this issue and to gamesindustry.biz for providing a forum to discuss it.

Posted:A year ago

#6

Anthony Chan
Analyst

88 72 0.8
Warren has it right, if games are to be considered an art. But I think the point we are all missing in this article and in our comments is Metacritic is not aimed as a way to judge a game for the developer/publisher. It is a way for the consumer to quantify its entertainment value relative to games that already exist. Thus, Metacritic does should not actually be a measure of a game's success, but rather a score to which the consumer may use to make an "educated" decision (buy, buy 2nd hand, rent, borrow, or maybe don't bother at all).

Posted:A year ago

#7
reviews and opinions are a tough subject because to be honest, every single person has one. Who is an expert when it comes to subjective matters of taste? no one is , that's the problem.

I look at it this way, if you were to say judge vanilla ice cream, you would have some that would say, " Yumm a taste that will be loved for centuries", But you'll also get " Yuck, vanilla sucks, chocolate rules", of course followed by the occasional' " Vanilla and chocolate are so played, tuity fruity is where its at", and of course the " Ice cream? who the hell likes ice cream?"

Which one is right? yeah, thats the problem. They all are.

Posted:A year ago

#8

Greg Wilcox
Creator, Destroy All Fanboys!

2,156 1,076 0.5
@Todd: Not really. If all one knows is crappy ice cream, when they DO have a taste of a brand that's superior, it often leads to a rethinking of what one "knew" to be the "best" (and makes for expanded horizons in the process).

Back to Metacritic, it's a total waste these days because aggregates work against a good game that might get knocked outside the score box some use to gauge a decent game (and this is another bone of contention, as one man's 70 is another man's 25, some ill-informed folks "ONLY buy" games that score an 80-85 or more on Metacritic and so forth and so on). If they just ran the damn links and didn't do any math, it might be a more trusted site in the future.

Posted:A year ago

#9

Eric Pallavicini
Game Master

280 180 0.6
Metacritic is like a ring of power, one place to collect them all (reviews) and in the number bind them all!

(And you don't even need to read the lenghty text on the precious, all you need is the shiny average.)

Edited 1 times. Last edit by Eric Pallavicini on 13th May 2013 9:54pm

Posted:A year ago

#10
@ Greg you miss the point, there is nothing crappy about the flavor of Vanilla ice cream, but that doesnt mean many people still wont care for it, or like it. Both views however can be valid and are valid.

The point is, the flavor of vanilla is loved by many, but at the same time many people dont care for it. So what is true then? is it a good flavor or bad? or both? if both, how do you score that?

Edited 5 times. Last edit by Todd Weidner on 13th May 2013 10:19pm

Posted:A year ago

#11

Greg Wilcox
Creator, Destroy All Fanboys!

2,156 1,076 0.5
Well, not liking something doesn't make it "bad" at all Todd. That's the point I'm making. You can't validate anything without trying it yourself and some people who use Metacritic averages strictly to judge games don't try as many of them as they should (I run into a few every so often and it just amazes me that they can't rent something {that's rentable} and have a taste.

Posted:A year ago

#12

Bruce Everiss
Marketing Consultant

1,692 594 0.4
When EA started benchmarking their output against Metacritic it led to a jump in the quality of their games. Dire products no longer made market (mostly).

If you do a publisher by publisher search on Metacritic you can see some awful stuff that some of them have wasted resources bringing to market. Total management ineptitude up there for all to see.

Posted:A year ago

#13

Gareth O'Neill
Environment Artist (Contract)

30 23 0.8
I think the thing that Annoys me the most about Meta critic is when a bonus for spending 2 or more years of your life working on a game is dependent on obtaining and sustaining a meta-critic score of over 80% for a min of a couple of months. Especially given the fact you're only working in one department of the game,you can give it your best, but if the other departments drag it down and don't pull their weight you're left bonus-less but not for lack of trying.

Posted:A year ago

#14

Gareth Eckley
Commercial Analyst

88 67 0.8
Artists and businessmen have differing priorities. Pictures at 11.

People seem to be glossing over the fact that Metacritic also encompasses the efforts of your advertising and business teams in co-promoting and increasing customer investment in the brand and is normalised by contemporary games released. If you launch a new FPS franchise against a new Halo product, you can expect to be marked by comparison.

Outliers aside, they do reflect a decent enough average opinion of a game's perceived quality, given the available data. It's not a perfect system, but then what do you expect when you attempt to compare "Dear Esther", "Amnesia", "Tribes: Ascend" and "Diablo 3" on the same scale?

Sure, it tends to cater towards the lowest common denominator, but complex and challenging games can still score well, provided they are presented so as not to drive away the intellectually timid.

Posted:A year ago

#15
The way I see it, you can use Metacritic as the bluntest crudest yardstick with which to benchmark your game development. After that, you can choose the fine line of throwing all your eggs and waffles to get a better benchmark, or throw caution to the wind and tweak your own personal way forward for your team. Just hope you make some good friends in the press along the way.

Posted:A year ago

#16

Jakub Mikyska
CEO

199 1,091 5.5
I think that the biggest problem of Metacritic is not what it does, or that its aggregated scores really do matter. The main problem is that nobody knows what's happening behind the curtain. As Warren wrote himself, some media is not even counted into the aggregated score, some media's influence is bigger than other's. But nobody knows why and how. We let an outsider decide about what's good and bad in our industry.

I think that for variety of reason, our industry needs something like Metacritic. But it shouldn't be a commercial site operated by someone, somewhere. It should be an initiative of our industry's members. With rules agreed by everyone, from Activision to indies. With "weights" (how much the individual score matters within the aggregated score) of each media publicly known and floating. Reviewers have a big influence over our industry and it should be only fair if our industry actually rated them back. When someone reviews your game, without even finishing it, that's bad journalism and it should be reflected. When someone writes deep, insightful and fun to read reviews, his or her voice should matter more. And everyone should know that.

Nobody questions ESRB or PEGI. There are fair and well-known rules. And if you disagree with their opinion, you can appeal. What our industry needs is a similar organisation that aggregates the scores and reviews, based on known and fair rules where the voices of the best journalists writing about games are heard, regardless of whether they write for IGN, or have a small blog, and the voices of amateurs are neglected.

It is kind of a crazy dream, but I believe that someone like Warren Spector actually has the authority to make something like that happen.

Posted:A year ago

#17

Eric Pallavicini
Game Master

280 180 0.6
When someone reviews your game, without even finishing it, that's bad journalism and it should be reflected.
I agree with that, though :

1. Compared to the film industry, it would be much more work than 1-3 hours, since some games actually offer hundreds of hours of gameplay and you can only see their full potential by spending that much time on them. I doubt games reviewers are going to do this since their are to produce a review according to some deadline (shortly after release). Though I saw some reviewers make a pre-review (or first impressions) and state it clearly and then update it or make a second "full" review afterwards.

2. There is no reviewing on reviewers, or critics of critics (would make no sense I agree, but still, sometimes I feel like it would be required). While reviews are commonly accepted as opinions this allow some liberty to write whatever they like in whatever way they like and based upon whatever depth of testing is deemed appropriate. No scientific or consistent method to be applied there, since at the end there is still the argument that its an opinion.
It is kind of a crazy dream, but I believe that someone like Warren Spector actually has the authority to make something like that happen.
Some students in Texas will have the opportunity to learn from Warren and I definitely hope that this is a first step in the right direction and more universities and also more iconic veterans of the industry will be involved in such projects as the Denius-Sams Gaming Academy, including reviewers attending to those courses. Additionally, I support your crazy dream in full, having experienced a lot of what you say in your post both as a gamer and professional (wrote to reviewers a few times to tell them what I thought was wrong with their reviews, though while they answered me a few times they did not change it or updated it).

Edited 5 times. Last edit by Eric Pallavicini on 14th May 2013 11:21am

Posted:A year ago

#18

Farhang Namdar
Lead Game Designer

75 47 0.6
Some games are just too long for reviews, how can you rate a 200 hour RPG in less than a week. But editors ask that of their writers and so you get incomplete inaccurate reviews.

Also you have those reviewers that wish they were game developers and they miss the point entirely as well, solid reviews come from gamers that can write. Unfortunately there are too few of those when you look at the entire industry.

So Metacritic is mostly based on rushed work: So if your game magazine, website or whatever is hip enough and generates enough clicks you'll be on metacritic! Although I am certain you can pay the right people to get metacritic certified ;)

But this is how the entire industry works, nothing new...

Posted:A year ago

#19

Emily Knox
Associate Designer

47 96 2.0
Really interesting piece, and some thoughtful comments too. It's strange to consider people aiming to achieve a milestone on Metacritic, with no way of knowing how your work will be received out of 10, which elements of your work are more important to the person assigning a score, and no transparent structure to show how that score is translated by Metacritic. Amalgamating a collection of subjective opinions leaves a watered-down result; that final number doesn't tell you a great deal about a game. But reading a detailed opinion (whether this is by a critic and/or a consumer) will tell you exactly where they believe you fell short and where you succeeded. On an individual level, being able to discuss, read and hear about your work is a far more useful tool for improvement than being left with an interpretive overall grade (can you imagine if a maths exam was marked by numerous individuals, then a third party weighted their scores differently, without telling you which results mattered more, to make a result?)

I try to think of Metacritic the same way I would Rotten Tomatoes, this is a tool for people to quickly see whether something is worth paying for. But condensing all those thoughts together for one all encompassing result is a meaningless tool to measure individual success, the idea of defining success for yourself is a comforting thought and something I'd never really considered before.
Edit: This reminds me of Obsidian's Metacritic score of 74 for Fallout: New Vegas being deemed a failure, while 75 was a pre-determined benchmark of success.

Edited 2 times. Last edit by Emily Knox on 14th May 2013 9:42pm

Posted:A year ago

#20
An enthusiastic "This game sucks!" or "The AI is bad" or "This game gets a 4 out of 5" tells me next to nothing I need to know.
I agree with allot of this article but not this. Games are technical by nature and so I expect a review to tell me two things. Firstly I want to be able to judge whether this is the sort of game I would enjoy. Secondly I want to know whether the technical aspects are up to scratch. I want to know if the "AI is bad" because that can be extremely important. This dual nature of games reviews, I feel, is often overlooked. Critics obviously should strive for objectivity but accepting that isn't always possible they should say whether they think something is good or bad and give their reasons so the reader can decide whether they agree. This is the same for critics of other media and for this aspect of the review a score and particularly score aggregation are outright unhelpful. But there are parts of a game that can and, I believe, should be given a score. Are the controls laggy, is the AI shoddy, does it crash often etc. This is no different to when a movie review says the sound levels are off or the use of 3D is headache inducing. Clearly boiling the whole of a game to one score and averaging it across many reviewers isn't helpful and creates problems as discussed in this article but to say these kind of technical considerations have no value to the gamer is wrong.

Edited 1 times. Last edit by James Hastings on 14th May 2013 11:59am

Posted:A year ago

#21

Sandy Lobban
Founder and Creative Director

314 206 0.7
I think you take metacritic with a pinch of salt if you're making a game that you know isn't going to appeal to everyone. However, I would say there is a point, probably around 75 percent that is probably a good indication that the game is doing some things right regardless of the occasional clueless comments you may incur from someone who's having a bay day.

Its a bit like trip advisor. Everyone looks at it for a general consensus, and in that respect it kinda works. As with all other industries, there are some things that are just a waste of money, and its good to highlight these I think.

I was dead against the rating of individual developers though, and emailed them to remove me from it when it was being talked about a couple of years ago. That didn't reflect the way games were made. Its rarely an individual pursuit. Plus it would have been like a carrot on a stick for the wordy, work shy rock stars that live amongst the development community.

I remember someone giving us a rating of 0 percent for Wipeout HD/ Fury at Studio Liverpool, simply because they hated one feature alone. The general consensus from everyone else was good though, and the decent score made that particular reviewer look slightly incompetent at his job.

Edited 1 times. Last edit by Sandy Lobban on 14th May 2013 12:59pm

Posted:A year ago

#22

Rick Lopez
Illustrator, Graphic Designer

1,269 941 0.7
I dont believe a game should be required to get a high metacritic score, however I dont think it should be irrelivant. Games like Beyond Good and Evil, Mirrors Edge, Darksiders didnt sell alot, but that didnt make them bad games, I own all 3 and enjoyed them thoroughly. I think that a game creator should be driven by how passionate he is about the idea for the game he wants to create. Otherwise it would never be created. If we are driven buy the times, the culture and what sells, we'd all be making first person shooters and that would suck. And I think the Meta Critic website offers substantial data on how people percive your product. And wether it sells well or not one should take the data into account when producing a new game. I dont think EA should stop making Mirrors Edge or UbiSoft not make a sequel to Beyond Good and Evil. They were excellent games, however in a world driven buy first person shooters its understandable why they didnt sell well. RPG's and games like Zelda, had there moment of glory, but now its first person shooters. I think the worst thing a game developer can do is produce a game beyond his ability, capacity or means. To spend 100million developing a game and expect it to sell 12million copies to break even is kinda crazy. If your company cant handle that financial burdan, I say produce a game with a smaller budget and hope for more realistic sales projections. This is what out did THQ, the uDraw tablet. They thought it was gonna be the next big thing, upped there sales projections and then couldnt handle the financial burden when it flopped. Millions of units unsold in warehouses. I cant even imagine walking into a warehouse with millions of uDraw tablets. Anyway, unless we want a world full of first person shooters, I belive game creators should be passionate about what they are doing, that usually produces a good game. Sales projections should be kept to a realistic and practicle stand point and maybe balance production on games between cash cow games, games that follow the current trends and go in tune to the results and studies done through R&D, and then use some resources to produce games that YOU are passionate about. Alot of times in any industry its taking risks that eventually produce good results. Comic book movies were not popular until IronMan and The Dark Knight. And just because the ones before werent as popular, they didnt stop making them, the kept on being made and now they enjoy the premium status they have now.

I think perserverence eventually yields good results. Lots of trial error and failures. But its that time that you actually nail it and get it right that makes it all worth it. And those results can be much better than simply creating another shooter and shooting for a high meta critic score.

Posted:A year ago

#23

Brian Smith
Artist

195 84 0.4
Theres nothing wrong or irrelevant about metacritic. From a consumer point of view it's a time saver and generally a good thing. When I make a game purchase I check metacritic first. I don't live or die by the aggregate score. I look at the score, I look at the number of reviews, I read some of the key high and low reviews and this meshed with other reviews helps make my decision.

As for it being used by the industry, fed back into the loop, defining marketing budgets and bonuses, well that the fault of those who use it, not the fault of metacritic. It's pushing the responsibility away from folk paid to make those decisions and onto something remote and blame-able.

I find it strange what Warren said about Metacritic opening up and being more transparent in order to make it more fair on the important things it influences when it just shouldn't be used for these things.

Also, when I look up metacritic I'm not looking for what level of success it's had for the folk that made it (sales). I'm looking for information that will help me on a purchase. It'd do me no good knowing that colonial marines sold to over a million, it was irrelevant to the games quality.

Posted:A year ago

#24

Rick Cody
PBnGames-Board Member

144 14 0.1
I think there's a simple answer. Months before release, Metacritic should let the publishers and developers create a company-wide review scale by allowing them to edit the weight of review scores from individual reviewers (even if they're within a larger entity, like IGN).

This wouldn't be the same Metacritic score seen by the general public, but it'd allow publishers and developers to more easily measure success.

Posted:A year ago

#25

Tim Carter
Designer - Writer - Producer

556 293 0.5
Game designers spend too much time pontificating on stuff.

The people with power, on the other hand, pat them on the heads and tell them to get along.

Game designers should form a guild and make demands if they really want to change things.

Posted:A year ago

#26

Marty Howe
Director

59 25 0.4
Hi what would a game designers guild do, and what things do you really want to change? Maybe just bullet point them?

As has been said before, Metacritic should serve only as a 'guide' Metacritic is not perfect (like everything else in life) but its the best we have. I always read Metacritic before a purchase (games are $80+ in Australia) to obtain information important for me, such as- 'this game has no jump', 'this game is only 4 hours long', 'the AI is dumb and won't notice you' etc Factual and relevant information.

Posted:A year ago

#27

Yiannis Koumoutzelis
Founder & Creative Director

361 204 0.6
Metacritic is a decent way for quickly figuring out before buying whether people like that game or not. It is not meant to be an in depth analysis or anything like that and i do not see any gamers anymore giving it as much weight as they used to a few years back.

The good thing about metacritic, is that it offers both professional reviewer and gamer score. You can find them all together there and make up your mind. Is it accurate? Definitely more accurate than reading one review! That is for a gamer who knows his way around reviews and understands, which are paid promotional reviews, which comments come from fanbois, haters etc. etc. The same applies for app store reviews.

Another point for discussion could be the fact that It is not just gamers deciding to buy a game or not, but also many studios hire based on one's metacritic average of all the titles they have worked on!

Posted:A year ago

#28

Jack Pochop
Studying Telecommunications

27 16 0.6
As a writer, I think I like idea of reviewers and critics taking on several pillars of definitional success. As you had noted, even a bad writer - a bad reviewer - can be beneficial to read. As it stands, I see several platforms who engage their readers with their own opinions, and set their reviews up as solid platforms for contrast. Things like Metacritic are an interesting idea, and fun to read, but something I've never relied too heavily on. It's a great source for individual reviews, but isn't a telltale sign, for me, whether or not a game is going to be good.

Posted:A year ago

#29

Adam Campbell
Associate Producer

1,154 941 0.8
Well, Metacritic isn't irrelevant, even if its only one way to measure the quality a product you are buying into.

Posted:A year ago

#30

Kevin Patterson
musician

185 99 0.5
I think most gamers that actually take the time to view metacritic, don't simply look at the average score.
I use metacritic when buying a game i'm unsure about, but as I mentioned above, i don't simply look at the average.
I take the time to scroll down through the reviews reading high score and low score reviews, and sometimes read some user reviews as well. I like metacritic because it's a one stop shop to read all about a game I may be thinking of buying, lots of reviews in one place, and I do read reviews linked there to my favorite trusted sites.

Posted:A year ago

#31

Login or register to post

Take part in the GamesIndustry community

Register now