Sections

Find out how to kick start your games industry career

Get Your Free Ticket Today

Steam alters review system, irritates indies

Storefront only allows copies it sold to be used in aggregate review score, hiding opinions of scammers, crowdfunding backers, bundle purchasers, and more

Steam has changed the way it handles review scores, and not everyone is happy about it. The company detailed the changes in a blog post yesterday, saying they are intended to improve the usefulness of the review score system and thwart attempts to manipulate a game's aggregated user review score.

"We know this review score has become a valuable shortcut for customers to gauge how well the game is matching customer expectations," Valve wrote. "But the review score has also become a point of fixation for many developers, to the point where some developers are willing to employ deceptive tactics to generate a more positive review score.

"The majority of review score manipulation we're seeing by developers is through the process of giving out Steam keys to their game, which are then used to generate positive reviews. Some developers organize their own system using Steam keys on alternate accounts. Some organizations even offer paid services to write positive reviews."

To limit the impact of such tactics, Valve is changing the way it calculates the overall review score number at the top of a game's product page. From now on, that score will only reflect reviews written by people who purchased the game through Steam directly. Anyone who activated the game on Steam using a product key--whether they got the key through a giveaway, bought a copy from a different online store, received it through a Humble Bundle or as a Kickstarter backer reward--will not have their feedback included in the review score. They can still leave reviews that will be visible individually on the game's store page, but they won't impact the game's top-line score.

Valve said it found "at least 160 titles" across Steam that show significant discrepancies between the percentage of positive reviews from those who redeemed Steam keys and those who purchased the title directly from Valve, but it also noted there are legitimate reasons why that could happen. As of this writing, Steam lists over 10,700 games for sale, which puts the number of games with glaring differences at about 1.5% of the total. The new rules will adjust the review score category for about 14% of games on the service.

A number of developers have taken to Twitter to express their concerns with the change. Death Ray Manta developer Rob Fearon said his game retained its "very positive" rating, but is still hurt by having the number of reviews determining that score reduced by about a third.

"Anyone who has crowdfunded loses reviews, anyone who has an existing support base through direct sales loses out, anyone who bundled early," Fearon said in a series of tweets about the move. "And I would not care except Valve have made user reviews such an integral part of their storefront that I have to care about them."

Larian Studios developer Kieron Kelly echoed that thought, saying, "The new Steam Review policy will hurt. As a Kickstarter dev, your most passionate fans are now silenced." Larian is releasing the Kickstarted Divinity: Original Sin 2 into Early Access this week.

Maia developer Simon Roth said direct sales, Kickstarter, and Indiegogo backers have accounted for 16,000 users who are now unable to influence the game's score on Steam.

"Maia's review score just dropped to 45%," Roth said. "Which means we are now unlikely to sell another copy. Time for me to give up probably."

Chris Dwyer, who works biz dev and PR for indie storefront itch.io, saw the change as a good time to remind developers of the dangers of having a single point of failure in their business plans.

"It is a vicious, vicious jungle so- idk, be safe and diversify your success and never take one aspect of digital sales for granted," Dwyer said. "I can at least promise you that itch.io is a safe and incredible space for your direct sales, that's the least I can try and help with. You can build rapport and community at @itchio and then you aren't at the whim of an unpredictable giant who always shoots first, asks later."

Find out how to kick start your games industry career

Get Your Free Ticket Today

More stories

Valve releases game compatibility details for Steam Deck

Company says the categorization of games with portable device will be an ongoing evaluation process

By Jeffrey Rousseau

Valve files for dismissal of Wolfire Games' antitrust lawsuit

Steam's parent company filed a motion to dismiss, saying its practices do not harm competition

By Marie Dealessandri

Latest comments (12)

Ben Link Video Game Enthusiast and Graphic Artist 5 years ago
I like the move. Any way of removing the doubt of a fake review score is okay with me.
6Sign inorRegisterto rate and reply
Sean Kauppinen Founder & CEO, IDEA5 years ago
The goal in this move seems to be more towards ensuring Steam is the only platform where you can buy a Steam key if you want the full benefits of being a part of the community. This affects Humble, GoG, Greenman Gaming, GamersGate, and every other digital store as well.

This is not good for the health of the ecosystem of independent game development. A more appropriate approach is verification within Steam's own algorithm (does the Steam user only post positive reviews, have they reached a legitimate threshold for credibility (like N4G or GiantBomb has), and does the user actually play the games they have for a minimum time period?), would be a more effective and fair approach to this.

If Steam is going to remove the most ardent backers of a project from the review system, it should also remove reviews from players that refund a game before the two hour limit for refunds, or at least flag the person as someone that has refunded the game.

Accounts with less than 5 reviews (arbitrary, but akin to needing to make at least one Steam purchase before reviewing today) shouldn't have their reviews show.

Patterns of commenting and reviewing with other like accounts giving 5 stars to the same games could be analyzed to weed out fraud.

This is also a really awful day for indie devs that rely on crowdfunding communities, as well as those that work with other stores to build momentum like Greenman Gaming and Playfire.

I still believe Steam is a fantastic platform, however this is not well thought out. I'm hopeful they find another way to deal with these issues.

Edited 1 times. Last edit by Sean Kauppinen on 13th September 2016 5:26pm

11Sign inorRegisterto rate and reply
Tyler Moore Game Designer & Unity Developer 5 years ago
By excluding reviews from people that may not have paid for the game, they're also excluding reviews from people that have an emotional investment which may bias their opinion whether they're aware of it or not. If the reviews solely reflect the people that bought it from steam, I would argue it's a better (but not perfect) reflection of the game's user base on Steam. As someone shopping on steam, that score is now a better measure of whether the game is worth it or not.

Maybe you're breaking some eggs by also excluding people that bought it from another source (G2A or direct from the developer), but I would argue that's a cost worth the benefits.
3Sign inorRegisterto rate and reply
Show all comments (12)
Greg Wilcox Creator, Destroy All Fanboys! 5 years ago
As much as I don't want to, I'm going to agree with Steam and also ask what the hell took so long? Seeing a game score Positive to Mostly Positive with the bulk of reviews written by freebie getters who've played under half an hour (or less) is galling. Especially when there are so many just plain BAD asset-lifted games bought and resold as-is or with wacky changes on Steam that they ought to force those "developers" to pass some sort of stricter quality control in the near future.

The one thing that NEEDS to be fixed is allowing bundle users to submit reviews. However, I'm gathering as Steam doesn't see much profit from non-Steam sales, they can "penalize" bargain seekers unless it's a Steam-curated bundle or sale. And yeah, they KNOW they can get a bunch of fresh reviews up from folks who buy during a Steam sale...
5Sign inorRegisterto rate and reply
Bonnie Patterson Narrative Designer, Writer 5 years ago
Hmm... some of the indies may not be very happy right now, but I suspect that will change rapidly the second they get an angry mob trying to lynch their product.

Plus, it does make sense - the reviews are on Steam, about Steam games and if they're all taking about the game as it's played on Steam, it means they all have a common base for comparison.
5Sign inorRegisterto rate and reply
Morville O'Driscoll Blogger & Critic 5 years ago
The goal in this move seems to be more towards ensuring Steam is the only platform where you can buy a Steam key if you want the full benefits of being a part of the community. This affects Humble, GoG, Greenman Gaming, GamersGate, and every other digital store as well.
It depends what you mean "full benefits... of the community". You can still write reviews for Steam Activations, you can still write guides, you can still comment on other people's reviews, you can still post in Discussions. The only thing you're missing out on is your review being part of the aggregate percentage scores. That's it.
A more appropriate approach is verification within Steam's own algorithm (does the Steam user only post positive reviews, have they reached a legitimate threshold for credibility (like N4G or GiantBomb has), and does the user actually play the games they have for a minimum time period?), would be a more effective and fair approach to this.
Yup, fully agree that these alternatives are somewhat better than the one-size fits all basis that Valve have done. That being said, none of these alternatives actually deal with the problem of biassed paid-for reviews, so it still wouldn't fix the core problem.
If Steam is going to remove the most ardent backers of a project from the review system, it should also remove reviews from players that refund a game before the two hour limit for refunds, or at least flag the person as someone that has refunded the game.
There are totally legitimate reasons for reviewing and refunding a game - the obvious being fundamental software and hardware issues which prevent playing or completing the game. But it should be obvious now that Valve could easily add a category of "Consumer Refunded Reviews" under the "Key Activations" to give a finer view of reviews. That they haven't done this yet is pretty shocking.
Accounts with less than 5 reviews (arbitrary, but akin to needing to make at least one Steam purchase before reviewing today) shouldn't have their reviews show.

Patterns of commenting and reviewing with other like accounts giving 5 stars to the same games could be analyzed to weed out fraud.
Neither of these are any better than the current iteration - I've reviewed 2 games on Steam, and have thousands of hours played across hundreds of games. And maybe someone who gives a thumbs-up to just RTSs just really likes RTSs and doesn't play much else?

The main issue is that this is done without thinking of the borderline review tiers - this has dropped some games from Very Positive to Mostly Positive, and other games from Mostly Positive to Mixed. That's a powerful change that warrants more than a cursory thought.
5Sign inorRegisterto rate and reply
Jordi Rovira i Bonet Lead Engineer, Anticto5 years ago
As a tiny unknown indie studio we like this move. Even if we are pretty irrelevant we get dozens of emails of people asking for a free code in exchange of a positive review, which we reject. We can't help to think that maybe not everbody does, though.
5Sign inorRegisterto rate and reply
Anthony Gowland Director, Ant Workshop5 years ago
excluding reviews from people that have an emotional investment which may bias their opinion whether they're aware of it or not
Emotion and subconscious bias come in to every single review anyway. Are you saying someone who kickstarted a game and loves it has a less valid opinion than someone who bought it months later and loves it?
it means they all have a common base for comparison.
Not really - patches and variable pricing over time mean there's plenty of change. And that's ignoring that people weight aspects of a game differently anyway.

If Valve are able to identify 160 games they think are messing with reviews, how about dealing with those people directly, since it's almost certainly against the terms of service for developers? It would be less dramatic than making sweeping changes to the store, that won't stop scammers from scamming anyway.
1Sign inorRegisterto rate and reply
Tyler Moore Game Designer & Unity Developer 5 years ago
@Anthony Gowland:
Emotion and subconscious bias come in to every single review anyway. Are you saying someone who kickstarted a game and loves it has a less valid opinion than someone who bought it months later and loves it?
Yes (to me subjectively at least), in the context of "What do other people who bought this game off steam without having invested in the game think?". I'd rather get my opinion of a product from someone who bought it off the shelf than an investor of that product. Hype culture is a real thing.


I Agree with your second point. It's tough to use reviews to gauge the value of a game when the price and quality fluctuate. I suspect Valve's "recent review" system has been an attempt to remedy that.

Edited 3 times. Last edit by Tyler Moore on 14th September 2016 5:19pm

3Sign inorRegisterto rate and reply
Craig Page El Presidente, Awesome Enterprises5 years ago
I like it, and not just because my own rating went from Mostly Negative to Mostly Positive. I think it just makes more sense...

Ask yourself, if you're browsing games on Steam, which reviews are more likely to match up with your own tastes? Reviews from other users who bought the game directly from Steam? Or outside reviews from people who got the game through Kickstarter, bundles, or giveaway sites?
1Sign inorRegisterto rate and reply
James Prendergast Process Specialist 5 years ago
@Craig Page: Wait a minute... Whose "reviews are more likely to match up with your own tastes"? This isn't facebook we're talking about here. We're not advocating echo chambers of reinforcement... User reviews should be able to reflect the whole gamut of user tastes - not just those that you or some algorithm might decide you to favour.

At the end of the day, the problem was never the reviews themselves, but the 'number'/'rating' - because the reviews themselves are not gone. How many discussions have we had about the ridiculous nature of assigning a number to a game? Steam's response is to cull what goes into the number/thumbs up_down?

As I understand it, discoverability is aided by being able to expand your 'game horizon'. I've done it many times. Genres I've never played before, game types that turn me off from their description or initial try-out.

IMO, the right decision is to remove numbers and positive/negative scores and just have the reviews. Let people read what people say with a date, hours played and version number. Reviews with lower version numbers are pushed towards the bottom. If a person can't decide for themselves outside of a number or a general positive or negative then maybe the game isn't for them in the first place.
1Sign inorRegisterto rate and reply
Nathan Richardsson Executive Producer 5 years ago
While I totally understand the problem that they are trying to solve, this is about the most blunt way of doing it. I mean this says it all:
"Steam lists over 10,700 games for sale, which puts the number of games with glaring differences at about 1.5% of the total. The new rules will adjust the review score category for about 14% of games on the service."
We had a Founder's program so those 900 reviews went away, we use Steamworks extensively so our retail boxes are also external keys and won't count and we have electronic re-sellers and those keys are also invalid.

First part of a cleanup like this would be systematically identifying accounts and games with large number of free keys and large number of reviews. It should give a pretty nice list of positives which you can apply further filtering on to catch the biggest offenders. You can even link them to the real accounts people have.

Secondly, reviews from 0.2 hours played is just silly. At least have a proper minimum because a very clear indicator of a paid reviews is played time. Not to mention that giving a meaningful review after 20 minutes is kind strange.

Third, of course a minimum playtime before reviewing should be in, and higher than the 2 hour limit of refund.

Fourth, you could allow reviews from refunded copies, low playtime etc. but not count them in the review score. I mean, if brute force removing hundreds of reviews is OK without even analyzing them, that must be considered fair as well.

Because right now, while Eternal Crusade isn't about to be the biggest game in the world, a number of fully valid resellers and physical sales worldwide just got screwed along with us. I hope I elaborated a bit on why this isn't just an indie problem, and certainly not a practical solution to deal with a 1.5% problem.
1Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.