Steam adds histograms to address review bombing

Ability to see how user reviews change over time one step on the way toward prediction-based review scores

Review bombs have become a common occurrence on Steam, and Valve is looking to change the way it handles them. In a post on the Steam Blog, Valve's Alden Kroll laid out the company's thinking on the practice, where players leave negative reviews for a game en masse in order to push the review score down and hurt its financial prospects.

"So why is review bombing a problem?" Kroll asked. "On the one hand, the players doing the bombing are fulfilling the goal of User Reviews - they're voicing their opinion as to why other people shouldn't buy the game. But one thing we've noticed is that the issue players are concerned about can often be outside the game itself. It might be that they're unhappy with something the developer has said online, or about choices the developer has made in the Steam version of their game relative to other platforms, or simply that they don't like the developer's political convictions. Many of these out-of-game issues aren't very relevant when it comes to the value of the game itself, but some of them are real reasons why a player may be unhappy with their purchase."

An example of Steam's new user review score histogram.

An example of Steam's new user review score histogram.

In the cases where the review bombers were reacting to an out-of-game issue, Kroll said the act usually caused a temporary distortion to a game's review score. But in cases where the scores didn't return to their prior level, Kroll said, "we believe the issue behind the review bomb genuinely did affect the happiness of future purchasers of the game, and ended up being accurately reflected in the regular ongoing reviews submitted by new purchasers." Whether developers changed their game in response to the review bomb didn't have any correlation to how their review score changed after the review rush was over, he added.

"In short, review bombs make it harder for the Review Score to achieve its goal of accurately representing the likelihood that you'd be happy with your purchase if you bought them. We thought it would be good to fix that, if we could do it in a way that didn't stop players from being able to voice their opinions," Kroll said.

The fix Valve settled on was to add a user review histogram on every game's main Steam page. The histogram charts a game's ratio of positive-to-negative reviews since its launch, and lets users click on a time period to see sample reviews from that time.

"As a potential purchaser, it's easy to spot temporary distortions in the reviews, to investigate why that distortion occurred, and decide for yourself whether it's something you care about," Kroll said. "This approach has the advantage of never preventing anyone from submitting a review, but does require slightly more effort on the part of potential purchasers. It also has the benefit of allowing you to see how a game's reviews have evolved over time, which is great for games that are operating as services."

The new information also needs to be put into slightly different context, as Kroll said review scores on most games generally trend downward over time as those who are most interested and likely to enjoy a game tend to purchase it closer to launch.

As usual with Steam, this tweak is not the final word on the subject, but one thing Valve is trying en route to another solution it has in the works.

"It's quite possible that we'll need to revisit this when we move to personalized review scores, where our prediction of your happiness with a purchase is based upon the games you've enjoyed in the past," Kroll said.

More stories

US District Judge says that Valve will face antitrust ligation

Judge Coughenour said that Wolfire Games' "allegations are sufficient to plausibly allege unlawful conduct"

By Jeffrey Rousseau

Will Steam Deck suffer from Valve's short attention span? | Opinion

Valve is building a track record of great hardware launches - but also tends to lose interest post-launch. Can Steam Deck break the streak?

By Rob Fahey

Latest comments (4)

Randal Woody Owner/ President of Digital Island Productions Inc 4 years ago
Pretty interesting! Trolls and haters are going to be out there. We cant get rid of them they are everywhere with an opinion and we cant just drop them to the bottom of the ocean where they belong! The child-like behavior can sometimes be remedied by growth into adulthood. But as we all can agree from what we have seen in today's political climate even adults can act like babies with a soiled diapers. I can honestly say I have given bad reviews on games in the past but I have also given praise. I have seen some game developers who had a great product ruin that said product by refusing to listen to their target customers. I do think that this histogram approach is a good idea. But still even with that implemented I would still be reluctant to buy a product with a low score. One solution I think would be cool is to allow more DEMO modes. Where we can download and play a demo version even with multiplayer and see firsthand if a game is good or bad. If it is bad just uninstall it and you have spent no money! This might open the door for hackers to CRACK the demos thus killing the developer.
0Sign inorRegisterto rate and reply
Nathan Richardsson Executive Producer 4 years ago
Agreed, there are other options and we're considering putting the free version simply as a demo. We've got a great amount of bombing, some less than 1 hour and usually a copy/paste from other reviews about a promise that was a pipe dream 5 years ago. I read them all.

So, just using common sense:

- At least require 10 hours play to submit a review. Press is held to much higher standards, I don't think to require 10 hours, is unreasonable. 0.7 hours and a review the person is just part of a bombing, not even mentioning the game, should not be a review.

- Use common practice of review systems. There are a bunch of review systems that work. Look at Amazon etc. where the weight of your review is based on history. In our case, people were logged in, while they wrote the review. Didn't play.

- Everyone pays millions to Steam for a service, that has agreements, such as terms of service and policies that is to protect my customers from other customers. If you pay 30% of revenue and get shit for pointing out - hey this is obnoxious, bodily harm - that it's his right to say he doesn't like the game. Where did he say reasoning for the game, besides wanting to kill a kitten with a piano wire? Even flagging people don't get acted upon. I had to go up the chain to get that review out. I still have reviews on my games flagged, nothing has been done.

- PUBG has paid Steam 20M+ (regional pricing kicks in), don't tell me that Steam/Valve can't have professional moderators or any of the companies that specialize in it, as part of those $20M. Just. That. One. Game. And don't try to say it's all going to bandwidth. I did product development on FTTH, xDSL and content distribution networks. They even pass on chargebacks, instead of having proper fraud protection and as part of the revenue share, protect us. Nope, we pay the fine.

Summing it up, still the fanboi but some of this is just not right.

Edit: Another opinion -

Edited 1 times. Last edit by Nathan Richardsson on 20th September 2017 11:37am

1Sign inorRegisterto rate and reply
Hugo Trepanier Senior Game Designer, Ludia4 years ago
At least require 10 hours play to submit a review.
Nathan, this really depends on the game; it would be way too much for several indie titles. I have played (and reviewed) many games that don't even offer 5 hours of play in total, so if you'd expect this much for the right to review then they'd have no score at all. Fewer games even offer under 1 hour of original content, so even establishing a very low minimum may be too strict.

If anything, a "review threshold" would have to be set by the developer on a per game basis. This could be a varying number of play hours, or simply reaching a certain section of the game (such as halfway through or at the very end), to ensure players have been exposed to the right amount of content.

What's missing on the graph is an indication of app updates on the timeline. It would be useful to see when changes in user opinion are tied to specific changes to the game.

At a glance I see this new graph tool as a nice toy for power users interested in numbers but I can't imagine the "general public" or average user to really care much for such granularity when deciding whether to buy a game or not.

Edited 1 times. Last edit by Hugo Trepanier on 20th September 2017 6:08pm

1Sign inorRegisterto rate and reply
Show all comments (4)
Cain Sinopoulos Community Developer, Massive - A Ubisoft Studio4 years ago
At least require 10 hours play to submit a review.
I agree with Hugo... this is an insane requirement. Yes, it could be setup on a per game basis, however; if this were to be done I believe that there should also be certain measures put in place to protect the player too.

For example. I think that if you're going to require someone to play your game for 10 hours before leaving a review, then your game should be eligible for an automatic refund up until that point PLUS the standard 2 hours Steam offers by default for its automatic refund policy. (10 hours to reach your requirement + 2 hour window in which they can leave a review)

If you're comfortable thinking that your game is good enough that someone should be forced to play it for the 10 hours needed to write a review, then there shouldn't be any issues allowing them to play long enough to review your game, tell you and others if they are not happy with it, then request a refund.

Existing systems are no doubt partially to blame as to why there are some many negative reviews with so little playtime (Not including review bombing). You're pushed into knowing that you don't like a game within the first 2 hours... so you have to jump in, get your settings the way you like them, setup your keybindings and what not. Then you need to get a feeling for the game a short amount of time which doesn't work for every game.

If the game is story or cut scene heavy you might not be able to get a good grasp of it in 2 hours. If a game has a long yet rewarding progressions system in it... you possibly can't experience that in 2 hours. It's no wonder that so many games have bad reviews when players can't even scratch the surface of the experience it offers.

I agree in these cases that a higher OPTION might be needed but if specific criteria are going to be added then there needs to be a balance between having enough time to get into the core game experience and not setting the limits so high that you're effectively silencing people with ridiculously high criteria... of course, making sure that the customer is protected and given the opportunity to share their thoughts while still having an option to refund their game.

If you're scared that players will simply play a bunch and refund the game anyway... well, they are already doing that. Just keep to the existing 2 hour requirement for your game (on Steam). If you're confident you've made a a gem of a product and think your game warrants 10 hours of play... well, that's your risk to take.

Edited 1 times. Last edit by Cain Sinopoulos on 21st September 2017 8:28pm

0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.