Sections

What drives a review score?

Sartori Bernbeck, head of EEDAR's Analytics team, examines which factors have the biggest impact on a reviewer's score - it may not be what you think...

During the course of a video game review, reviewers critique a broad array of game design factors: from the main character's clothes to the extensiveness of player agency and choice within a story. But which factors do reviewers spend the most time discussing, and which truly drive the final score?

In order to better understand reviewer behavior and analyze which aspects of video game design drive review score, EEDAR's research team collects and dissects console video game reviews across many different platforms and genres. Once reviews are collected, the research team analyzes the review text and identifies the specific positive and negative callouts the reviewer makes about the reviewed title. These callouts are categorized as one of 63 unique game aspects that fall under 8 primary categories: Gameplay, Market, Social, Narrative, Graphics, Technical, Value, and Audio. These categories focus on the following gameplay aspects:

EEDAR tracks the following data for each review callout: Callout Text Length, Callout Sentiment (Positive or Negative), Callout Aspect, and Callout Category. EEDAR uses this review database to help game developers better understand what factors have strongly influenced past reviews for titles comparable to their own. In this article, EEDAR shares three takeaways from the review data.

1. Reviewers focus most on Gameplay, but they also critique a broad range of other game aspects.

One of the keys to understanding what drives reviewer sentiment is simply understanding what topics reviewers spend the most time discussing. The following pie chart illustrates the average amount of text within a review allocated towards discussing each category.

Gameplay discussion receives the largest allocation of critical review text, as reviewers tend to be very descriptive and explanatory when discussing how a game functions. Within this category, callouts explaining mechanics and the feel of a game's combat systems account for the largest amount of text.

Across the seven other categories, reviewers direct a more even spread of attention. While reviewers focus more strongly on categories they feel are important (such as Market and Social), categories such as Value and Audio still receive an appropriate amount of callout text to discuss their strengths and weaknesses.

2. Narrative Aspects in games remain a pain-point in reviews.

Another interesting application of the review database is to understand general game factors that the reviewer community broadly feel games are "getting right." EEDAR looked at the percentage of callouts that were positive for each category. Based on this approach, the graph below shows areas reviewers feel games are more often succeeding than not.

Across reviews, graphics are discussed in a mostly positive light. Reviewers also have generally positive things to say regarding social elements, audio, and gameplay features. While games continue to provide strong Graphics and Social offerings, titles that do not differentiate from current market leaders are heavily scrutinized (noted within the Market category). Market is the second-most discussed topic with generally negative sentiment.

Narrative and Technical are areas that have grown more negative in recent years. Narrative Aspects in games are mostly criticized and placed against high standards in reviews. Reviewers often feel that the advancement of Narrative quality has been lagging behind that of other key game aspects such as Gameplay, Social, and Graphics. Bugs, connection problems, and other aspects within the Technical category have been growing in frequency for new releases, leading to a growing negative sentiment.

"Even for titles where multiplayer is a key staple of the experience, overall sentiment in reviews is primarily driven by the story-based campaign experience"

Review callouts for Graphics being very positive demonstrates not only development teams' success in creating visually appealing titles, but it also speaks to the diminishing returns graphical improvements have been seeing in recent years (as reviewers tend to be satisfied by most games' graphical achievements). This signifies an opportunity for developers to start shifting more focus onto other key areas of development such as Gameplay, Narrative, and Technical Aspects, which are categories that receive more mixed overall sentiment in reviews.

3. Single-player ultimately drives review score, even for titles with strong focus on multiplayer.

So if Reviewers cover a broad diversity of game factors and generally feel that graphics are often done well while narrative is lacking, what review factors actually impact the score of a specific game the most? EEDAR combines total text allocation with a rating of sentiment severity in order to more accurately understand how that point of discussion is impacting the final review score across a set of games. The strength of impact for each category was indexed using Gameplay as a base.

Market factors (i.e. comparisons to market leaders, comparisons to the franchise's previous iterations, and how well the title lived up to its own expectations) are the strongest drivers in impacting review score. These discussions have twice the amount of impact in review score over general Gameplay discussions. The game's plot is also crucial to review score; Narrative progression holds a strong weight across all titles that include a story mode.

Despite being the third-most discussed subject (13 percent), multiplayer elements (Social) ultimately hold a low impact in review score. Despite significant multiplayer features in titles such as Assassin's Creed Unity, reviewers tend to discuss multiplayer features in a more explanatory fashion without strong opinion. The likely explanation for Social's lack of importance is that most reviewers play a title before it is released and without an active community. This does not allow the reviewer to fully experience the quality of multiplayer features.

Even for titles where multiplayer is a key staple of the experience, overall sentiment in reviews is primarily driven by the story-based campaign experience. For example, the campaign for Call of Duty: Advanced Warfare was found to be significantly more impactful than the multiplayer elements for review score.

Unsurprisingly, Call of Duty: Advanced Warfare's Combat Systems (with the inclusion of the new EXO suits) played the most crucial role for reviewer sentiment. This aspect was found to be 2.24 times more influential in reviews than the 10th ranked aspect, Quick Time Events. The game's story mode was often discussed: aspects such as Campaign, Characters, Dialogue, and Onboarding Process were found to be very influential in reviews. Conversely, the only dedicated Social aspect found to be within the top 10 most influential aspects was Non-Campaign Co-op (the EXO Survival mode). Aspects such as Skirmish, Matchmaking, and Multiplayer Mode Diversity were not within the top 10 most influential aspects.

As multiplayer and general social offerings continue to grow in importance across games, it is important for developers to be aware that reviewers still primarily focus on the single-player experience. Currently, the best way to incentivize reviewers to focus more on multiplayer offerings is by providing the game for review once there is an active community playing the title.

Since a high review score is important for generating unit sales, it is crucial for developers to understand which factors ultimately drive review score. The importance of individual game aspects changes by genre and even by franchise, which makes it even more important for developers to understand how reviewers view their title within the context of the overall market.

Latest comments (11)

Rui Martins Senior Software Developer 4 years ago
For this study to have any real value, it must document all its assumptions, and all its rules used to classify Reviews texts and must be available for reference.
First there is huge assumption that what the review writes is directly correlated with what really made him score the game as he did. May seem a logical supposition, but it must be proved that it exists and is valid.
Another important factor, is the interpretation of EEDAR own reviewers, that review and classify game reviews. Do they interpret the same review in the same way (just another source of random noise, in the sample).
"EEDAR tracks the following data for each review callout: Callout Text Length, Callout Sentiment (Positive or Negative), Callout Aspect, and Callout Category."
How is "Callout Aspect" scored or categorized for example ?
Is it always same for their own reviewers ?

Writing can be changed to adapt to your audience (what your audience prefers to read), or might be hindered by the reviewer knowledge of the subject.
A reviewer with a music education knowledge background, might be more picky about game music score, or with a background in computer graphics, or anything else.
What I mean is that previous knowledge and experiences, i.e. the person background will certainly have more impact on how he reviews a game, than anything else. If a reviewer does not specially like a specific genre it will be difficult, if not impossible to provide an unbiased rating score. But if they avoid to rate these genres, than pnly reviewers that like that genre will rate it, and that will be biased too.

It also didn't seem logical, to suggest that developers should use this data to adapt their games for a better review score:
" ... it is crucial for developers to understand which factors ultimately drive review score."

Should developers develop games for reviewers or be concerned with their actual users/gamers ?

So all these "facts" must be taken with a grain of salt.
There might be some value in these findings, but it's hard to determine which are real facts from correlated data, and which are errors from input processing or presumed assumptions.

Edited 2 times. Last edit by Rui Martins on 9th February 2015 5:40pm

4Sign inorRegisterto rate and reply
Renaud Charpentier Game Director, The Creative Assembly4 years ago
Now what would be really interesting is the same split analysis but looking at the influence of reviews on sales.
4Sign inorRegisterto rate and reply
Charles Herold Wii Games Guide, about.com4 years ago
I agree with Rui. I spend a lot of time in my reviews discussing narrative in games (if there is any), because I really like stories, and they're fun to write about, yet when I'm scoring the game, I rate it mainly on gameplay. A great story might make me rate something a little higher, an awful story *maybe* slightly lower, but how much I write about the story rarely correlates much with my scoring decisions.
0Sign inorRegisterto rate and reply
Show all comments (11)
Jordan Lund Columnist 4 years ago
Speaking solely for myself, I approach each game with the idea that it's a perfect 10. I then play it looking for things that detract from the score or add back in to the score. Very rarely I encounter game breaking bugs that completely tank the review regardless of the graphics and story. For example: Assassin's Creed on the Vita shipped with a bug that corrupted the save game setting. After re-starting the game three times, I realized I couldn't give an honest review based on gameplay/graphics/story because unless you were willing to play it all in one sitting, it was literally impossible to play. Unacceptable for a full retail release. It was eventually patched a month later.
0Sign inorRegisterto rate and reply
Rogier Voet IT Consultant 4 years ago
It's simple a great story can make a good game better, but a great story wil not compensate for poor gameplay.
2Sign inorRegisterto rate and reply
Greg Wilcox Creator, Destroy All Fanboys! 4 years ago
Eh, these days I write reviews out of appreciation for fine work done in transporting me to a place I want to visit again and to help developers who trip up in spots make better games in the future.

I think there's a slightly backwards mentality in some writing about visuals in that some reviewers don't have an appreciation for stylization and instead mistakenly compare graphics based on what's most "realistic." Those types often refuse to see something like a "retro" 2D or polygon game as "acceptable" to their eyes, which is a shame when they dock a game for not looking like something running on a high-end PC with all the settings maxed out. *Yawn*

Don't get me wrong, I love my eye candy games. But when that's all some title have to offer and the gameplay is stumbly and patch-worthy right out of the gate, I'll hold out until enough fixes come in that make the experience what it was intended to be.

Story matters, but if it's an arcade-style throwback or other game where you just turn your brain off as soon as you hit Start, that's not a negative at all if that's the intent. Sometimes mindless fun is a perfect way to close out a day when you've been hunched over a keyboard spilling grey matter all over it.
1Sign inorRegisterto rate and reply
Andrea Schwendimann Editor / Author, Everyeye.it4 years ago
I agree on what Rui said.
I also find this article interesting, but a bit misleading in my opinion, when it comes to conclusions.

Especially on what you call "social" aspects, aka multiplayer.
From personal experience, first of all, a multiplayer mode in certain games has more value for players than in other games. Think about comparing Evolve with Mass Effect 3 multi. The latter is of much less significance for whoever plays Bioware finest sci-fi, hence, even if it was really limited despite beijg good at what it wanted to be, nobody really cared. On the other hand with something like Evolve, Dota 2 or Elite: Dangerous, we are talking of a totally different situation and value for players. Did the article and the data weigh this?

Second, CoD is not a good example for several reasons, in my opinion. I could say much about it, but I think the general feeling I share with my collegues (I can talk about my own only, of course, even though it seems to me that the impression is genuine for several other reviewers) is that the series stalled and needs a fresh start or some new twists, both multiplayer and singleplayer-wise.


Anyway, beside this, It is too often extremely difficult to evaluate the multiplayer for the press because, as said, we play games before release. This presents three typical scenarios:

1) the servers are simply empty or with few rare collegues trying hopelessly to figure out how to start a game without waiting for hours. This is the most common scenario.

2) some kind developers are online and you can either, A - immolate yourself to a carnage against one of them, which is not really useful or B - co-op with them, which is useful for a start and only to a point, since you must really play along with other "genuine" players to see if it is a good multiplayer or not. Otherwise you may fall under the impression to be either too guided or that that particular multiplayer mode is too easy or shallow, since its exploitable mechanics are thrown out in your face. This is a mild scenario, and often the one I prefer, given the other 2 alternatives.

3) The worst case scenario for multiplayer. A special event to try out the game, valid for a review. The "three days of the Controller", I like to call them. You are asked straight forward, with no exceptions, to review the multiplayer of a specific game (in which is usually an important game mode to cast such an event) and of course give it a specific score. We play until dusk for two-three days among collegues, we get to know each other in jolly cooperation and mayhem. Everything seems fine. Excpet that the REAL problem here is that we play via LAN. This has led the critics (including myself), to write about something perceived in a real different way compared to the biblic servers gehenna that is a "day-1" for online titles such as Battlefield 4 and the likes. The solution? Not an easy one, but not the point here either.

I would just like to know if EEDAR took all this into account when stating, "Single-player ultimately drives review scores, even for titles with focus on multiplayer".

Because, from my experience, that is either necessary or, simply, not true.

Edited 6 times. Last edit by Andrea Schwendimann on 9th February 2015 7:01pm

0Sign inorRegisterto rate and reply
Sartori Bernbeck Senior Analyst, EEDAR4 years ago
Hi Rui,

While outside the scope of this article (due to the depth and scale of the sources and methodology behind this), we are always very open and communicative with our clients when we run analyses such as these for their titles, and more than happy to go through our process and methodology over the phone and/or via email.

About your second point - the EEDAR research team tracks 63 unique game Aspects (such as Dialogue, Level Design, Combat Systems, etc.) which fall into the 8 distinct Categories shown in this article. As the research team reads through the review text, they mark callout text (where the reviewer notes positive or negative value). The callout text is tagged and associated with the Aspects it is discussing. Our research team has stringent definitions on how everything gets categorized, and all researchers are taught to categorize text along the same rules.

The purpose of this article was to show which topics are most commonly discussed in reviews and which ones bear more impact in review scores (given callout text length and severity of words used). The assumption of this analysis, as you brought up, is that reviewers are indeed discussing the topics that are influencing their final scoring. For the sake of properly informing their readers of their sentiment on a title, we believe this should hold true on a broad sense, even if there are individually skewed reviewers. On a broad level, it wouldn't make much sense if reviewers were lowly/highly scoring a title for reasons unsaid, if only for the confusion it would lend to their readers. We take a broad sample of reviews in order to best avoid sample bias from certain outlets and reviewers. We do categorize each review by outlet and reviewer, however, so that level of granularity is also available when analyzing specific franchises or genres.

The ultimate goal of this article was to raise awareness of which areas reviewers are really focusing on during their review, which is the text that their readers will be viewing and listening to for advice on game purchases. Since review score is known to be a strong factor in generating unit sales, it is important to understand how reviewers will perceive your title. Proper messaging and reception around a title's release (of which review score is part of) is important in generating positive buzz and sales amongst consumers.

We do agree that developers should be focusing their design decisions based on the consumer experience, especially since social media and online sharing are a growing source of consumer awareness on the quality of a game. We hope that this reviewer data can be used to help developers better understand the reviewer's angle within the overall release reception of a game. Preemptively understanding the likely angle press will dissect your game from allows the developer to be properly prepared for that part of the game's release reception and buzz. What the developer chooses to do with that knowledge comes down to choice (review embargo timing, messaging, development decisions, etc.).
2Sign inorRegisterto rate and reply
Sartori Bernbeck Senior Analyst, EEDAR4 years ago
Hi Andrea,

Certainly, multiplayer offerings and their importance vary strongly between franchises (and even releases within the same franchise). For titles such as Mass Effect 3 and the Assassin's Creed franchise, we would expect multiplayer features to take a backseat to single player offerings in reviews. Even for titles such as these, understanding the weight reviewers will likely place on multiplayer features only proves to provide developers with more knowledge before their game's release.

Call of Duty was chosen due to its strong importance on both campaign and multiplayer offerings. For a title that is played by millions and known for being a definitive multiplayer experience, it is interesting to note the importance that these multiplayer features ultimately hold in the title's reviews (versus, for example, what the consumers feel). The point of this article (and of our larger data resource) is to help developers better understand the angle that the press analyze titles. Positive buzz and press around a title's release date is important, and reviewers play a strong role in that. Understanding the reviewer angle will hopefully allow developers to strategize more effectively around their game's release (in terms of messaging, working with reviewers, press, etc.)

Regarding the difficulties of reviewing multiplayer features in reviews - the three points you brought up were spot on. These are very strong reasons behind why it appears that reviewers don't hold as much review weight in multiplayer. As titles become more social and focused on connected, multiplayer experiences, it will be important for both developers and reviewers to evolve the reviewing process and evaluate the game's features properly.

This difficulty in properly addressing multiplayer features in reviews is a reason why reviews for online social titles (such as MMOs, or League of Legends when it was first released) often have to be updated periodically to reflect the growth and impacts of the multiplayer elements. Dota 2 was lucky to have a very large open beta community for over a year before the title was officially "released", so reviewers had a strongly built up community to review the game with. Of course, the above article is not meant to address these titles that do not have single player campaigns. That is another field to address with even more variables for concern (such as ongoing game updates that render reviews out of date) outside the scope of this article.
1Sign inorRegisterto rate and reply
Michal Korec Editor/Analyst 4 years ago
Meanwhile in UK... Eurogamer has dropped review scores.
2Sign inorRegisterto rate and reply
Patrick Walker VP, Insights, EEDAR4 years ago
It will be interesting to see how that move works out for Eurogamer from a business standpoint.

My hypothesis is that while the move may make sense from a content or journalism standpoint, it will lead to a loss in page views as a percentage of the readers will go elsewhere for what they want - the quickly digestible evaluation of a game that a score provides.
1Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.