Steam alters review system, irritates indies
Storefront only allows copies it sold to be used in aggregate review score, hiding opinions of scammers, crowdfunding backers, bundle purchasers, and more
Steam has changed the way it handles review scores, and not everyone is happy about it. The company detailed the changes in a blog post yesterday, saying they are intended to improve the usefulness of the review score system and thwart attempts to manipulate a game's aggregated user review score.
"We know this review score has become a valuable shortcut for customers to gauge how well the game is matching customer expectations," Valve wrote. "But the review score has also become a point of fixation for many developers, to the point where some developers are willing to employ deceptive tactics to generate a more positive review score.
"The majority of review score manipulation we're seeing by developers is through the process of giving out Steam keys to their game, which are then used to generate positive reviews. Some developers organize their own system using Steam keys on alternate accounts. Some organizations even offer paid services to write positive reviews."
To limit the impact of such tactics, Valve is changing the way it calculates the overall review score number at the top of a game's product page. From now on, that score will only reflect reviews written by people who purchased the game through Steam directly. Anyone who activated the game on Steam using a product key--whether they got the key through a giveaway, bought a copy from a different online store, received it through a Humble Bundle or as a Kickstarter backer reward--will not have their feedback included in the review score. They can still leave reviews that will be visible individually on the game's store page, but they won't impact the game's top-line score.
Valve said it found "at least 160 titles" across Steam that show significant discrepancies between the percentage of positive reviews from those who redeemed Steam keys and those who purchased the title directly from Valve, but it also noted there are legitimate reasons why that could happen. As of this writing, Steam lists over 10,700 games for sale, which puts the number of games with glaring differences at about 1.5% of the total. The new rules will adjust the review score category for about 14% of games on the service.
A number of developers have taken to Twitter to express their concerns with the change. Death Ray Manta developer Rob Fearon said his game retained its "very positive" rating, but is still hurt by having the number of reviews determining that score reduced by about a third.
"Anyone who has crowdfunded loses reviews, anyone who has an existing support base through direct sales loses out, anyone who bundled early," Fearon said in a series of tweets about the move. "And I would not care except Valve have made user reviews such an integral part of their storefront that I have to care about them."
Larian Studios developer Kieron Kelly echoed that thought, saying, "The new Steam Review policy will hurt. As a Kickstarter dev, your most passionate fans are now silenced." Larian is releasing the Kickstarted Divinity: Original Sin 2 into Early Access this week.
Maia developer Simon Roth said direct sales, Kickstarter, and Indiegogo backers have accounted for 16,000 users who are now unable to influence the game's score on Steam.
"Maia's review score just dropped to 45%," Roth said. "Which means we are now unlikely to sell another copy. Time for me to give up probably."
Chris Dwyer, who works biz dev and PR for indie storefront itch.io, saw the change as a good time to remind developers of the dangers of having a single point of failure in their business plans.
"It is a vicious, vicious jungle so- idk, be safe and diversify your success and never take one aspect of digital sales for granted," Dwyer said. "I can at least promise you that itch.io is a safe and incredible space for your direct sales, that's the least I can try and help with. You can build rapport and community at @itchio and then you aren't at the whim of an unpredictable giant who always shoots first, asks later."
This is not good for the health of the ecosystem of independent game development. A more appropriate approach is verification within Steam's own algorithm (does the Steam user only post positive reviews, have they reached a legitimate threshold for credibility (like N4G or GiantBomb has), and does the user actually play the games they have for a minimum time period?), would be a more effective and fair approach to this.
If Steam is going to remove the most ardent backers of a project from the review system, it should also remove reviews from players that refund a game before the two hour limit for refunds, or at least flag the person as someone that has refunded the game.
Accounts with less than 5 reviews (arbitrary, but akin to needing to make at least one Steam purchase before reviewing today) shouldn't have their reviews show.
Patterns of commenting and reviewing with other like accounts giving 5 stars to the same games could be analyzed to weed out fraud.
This is also a really awful day for indie devs that rely on crowdfunding communities, as well as those that work with other stores to build momentum like Greenman Gaming and Playfire.
I still believe Steam is a fantastic platform, however this is not well thought out. I'm hopeful they find another way to deal with these issues.
Edited 1 times. Last edit by Sean Kauppinen on 13th September 2016 5:26pm
Maybe you're breaking some eggs by also excluding people that bought it from another source (G2A or direct from the developer), but I would argue that's a cost worth the benefits.
The one thing that NEEDS to be fixed is allowing bundle users to submit reviews. However, I'm gathering as Steam doesn't see much profit from non-Steam sales, they can "penalize" bargain seekers unless it's a Steam-curated bundle or sale. And yeah, they KNOW they can get a bunch of fresh reviews up from folks who buy during a Steam sale...
Plus, it does make sense - the reviews are on Steam, about Steam games and if they're all taking about the game as it's played on Steam, it means they all have a common base for comparison.
Neither of these are any better than the current iteration - I've reviewed 2 games on Steam, and have thousands of hours played across hundreds of games. And maybe someone who gives a thumbs-up to just RTSs just really likes RTSs and doesn't play much else?
The main issue is that this is done without thinking of the borderline review tiers - this has dropped some games from Very Positive to Mostly Positive, and other games from Mostly Positive to Mixed. That's a powerful change that warrants more than a cursory thought.
Not really - patches and variable pricing over time mean there's plenty of change. And that's ignoring that people weight aspects of a game differently anyway.
If Valve are able to identify 160 games they think are messing with reviews, how about dealing with those people directly, since it's almost certainly against the terms of service for developers? It would be less dramatic than making sweeping changes to the store, that won't stop scammers from scamming anyway.
Yes (to me subjectively at least), in the context of "What do other people who bought this game off steam without having invested in the game think?". I'd rather get my opinion of a product from someone who bought it off the shelf than an investor of that product. Hype culture is a real thing.
I Agree with your second point. It's tough to use reviews to gauge the value of a game when the price and quality fluctuate. I suspect Valve's "recent review" system has been an attempt to remedy that.
Edited 3 times. Last edit by Tyler Moore on 14th September 2016 5:19pm
Ask yourself, if you're browsing games on Steam, which reviews are more likely to match up with your own tastes? Reviews from other users who bought the game directly from Steam? Or outside reviews from people who got the game through Kickstarter, bundles, or giveaway sites?
At the end of the day, the problem was never the reviews themselves, but the 'number'/'rating' - because the reviews themselves are not gone. How many discussions have we had about the ridiculous nature of assigning a number to a game? Steam's response is to cull what goes into the number/thumbs up_down?
As I understand it, discoverability is aided by being able to expand your 'game horizon'. I've done it many times. Genres I've never played before, game types that turn me off from their description or initial try-out.
IMO, the right decision is to remove numbers and positive/negative scores and just have the reviews. Let people read what people say with a date, hours played and version number. Reviews with lower version numbers are pushed towards the bottom. If a person can't decide for themselves outside of a number or a general positive or negative then maybe the game isn't for them in the first place.
First part of a cleanup like this would be systematically identifying accounts and games with large number of free keys and large number of reviews. It should give a pretty nice list of positives which you can apply further filtering on to catch the biggest offenders. You can even link them to the real accounts people have.
Secondly, reviews from 0.2 hours played is just silly. At least have a proper minimum because a very clear indicator of a paid reviews is played time. Not to mention that giving a meaningful review after 20 minutes is kind strange.
Third, of course a minimum playtime before reviewing should be in, and higher than the 2 hour limit of refund.
Fourth, you could allow reviews from refunded copies, low playtime etc. but not count them in the review score. I mean, if brute force removing hundreds of reviews is OK without even analyzing them, that must be considered fair as well.
Because right now, while Eternal Crusade isn't about to be the biggest game in the world, a number of fully valid resellers and physical sales worldwide just got screwed along with us. I hope I elaborated a bit on why this isn't just an indie problem, and certainly not a practical solution to deal with a 1.5% problem.