Steam Spy and the need for numbers

In recent weeks it has become clear that Valve does not intend to let Steam Spy continue publishing Steam game sales data estimates unimpeded, leaving an information void that the company may fill itself...eventually. The industry snapshot that Steam Spy provided was illuminating, but for small developers and the publishers who work with them, its data often meant the difference between a scrapped project and a successful release.

Steam Spy launched in 2015, offering data that covered copies sold, users, average review scores, and plenty of other metrics, with a surprising if imperfect degree of accuracy. In April, an update to the Steam web API affected the way Steam Spy pulled data from the platform, forcing the service to temporarily shut down. It has since returned, though with its accuracy further reduced and the looming threat of Valve shutting the service down via another update at any time.

Since then, Valve representatives have commented on Steam Spy, citing concerns about its accuracy and going so far as to mention that Valve had some other tool in the works to replace it. But what that tool might be and whether it would provide equitable or better data than Steam Spy is still a mystery.

The blow to Steam Spy wasn't so much an affront to its creator, Sergey Galyonkin, as it was to the community of developers that relies on the data it presented. Galyonkin said in an interview with GamesIndustry.biz after the site closed in April that he had seen it coming, but that the removal of Steam Spy was a restoration of the information asymmetry that once allowed those who had access to metrics to abuse that power against those who did not.

"[Steam Spy has] made a material difference in many developers' lives, especially when it helped them secure funding without which they couldn't have made their games at all"

Lars Doucet

Publishers and developers with enough money to pay for the kind of data Steam Spy provided for free would once again be able to race ahead of small teams who had found the playing field leveled by the service. In fact, as a small developer himself, that audience was precisely why Galyonkin created Steam Spy in the first place.

"In my experience developers are using Steam Spy mostly for comps, to prove their idea for a new game has a chance of succeeding because there were other similar games before," Galyonkin said. "I also see developers looking into what languages to add first, what markets to focus on, and checking which similar games to play to understand what works and what does not."

Like Galyonkin, developer Lars Doucet has seen firsthand how numbers, imperfect though they may be, can help small developers. In 2012 (before Steam Spy existed) he published a blog post that included sales stats for his recent release, Defender's Quest, as well as other data. In response, multiple developers reached out to tell him how much that info helped them as a reference point in pitches to get their own games funded.

"I've seen people remark that sales stats aren't useful for anything other than curiosity or voyeurism," said Doucet, "but as we've been seeing, it's made a material difference in many developers' lives, especially when it helped them secure funding without which they couldn't have made their games at all."

Unfortunately, Doucet was forced to stop publishing numbers for his games. Valve eventually changed the terms of service for developers on its platform, prohibiting publication of exact figures in the manner Doucet had been sharing them. The reasoning was that these numbers could be cross-referenced with chart positions to reveal numbers for other games that had not been disclosed by their developers.

Steam Spy filled the void in 2015, and its importance to small developers was demonstrated by Doucet through a recent Twitter thread. In it, he asked for stories of those who had cited either developer-provided statistics or Steam Spy numbers in a pitch to get a game funded. The thread quickly filled with replies; all success stories.

One classic example came from Maged 'Squidly' Hamdy of Octosoft. The studio's game, Renaine, was just recently fully funded on Kickstarter, finishing the campaign with $22,307 pledged of a $5,000 goal. But that alone wouldn't fund the entirety of Renaine, and a chunk of money was needed to even get the game to its Kickstarter success in the first place.

Renaine failed its first attempt at a Kickstarter in 2017, but a grant allowed Octosoft to hire more team members and up the quality of its pitch. And even prior to that, an initial funding budget was necessary just to get the concept off the ground. To get that funding, Hamdy had to prove the concept was sound, and to do that, numbers were needed.

"I cannot overstate how important SteamSpy is, indies need to band together and try to get Valve to stop trying to keep us in the dark"

Maged Hamdy

"We were pitching for a Shark Tank-like setup at my university, and we essentially had to prove the market was there so we could formulate a business plan," Hamdy said when asked for more details about his account in the Twitter thread. "We did get funded, but it's safe to say without any sort of figures, which are extremely hard to find for indies without Steam Spy, we'd have failed to get any real funding, and Renaine would not have been a thing.

"I cannot overstate how important Steam Spy is. Indies need to band together and try to get Valve to stop trying to keep us in the dark. It's not perfect analytics, but the other option is absolutely nothing."

Similarly, Paige Marincak of Atemly Games used Steam Spy numbers in applying for grants to fund her game, Gataela.

"[The data] was mainly used to help support that there was a market for RPGs, specifically indie pixel art ones," Marincak said. "The data was also helpful in filling in the details that more general reports and information I had seemed to lack. It was quite invaluable in terms of seeing how a similar game did on the market and proving that RPGs were not, uh, 'dead,' as one person put it."

For Marincak, the data was helpful to get the grants she needed. But more than that, it helped legitimize what she was doing, even when other grants she applied for were not approved.

"It allowed me to at least get my foot in the door, instead of being outright shut out," she said. "As helpful as SteamSpy was for grants and applying for them, I personally found it more helpful in keeping me grounded and realistic about my expectations for not only Gataela, but all future games."

"Without the hard numbers from Steam Spy...I wouldn't be where I am today; a fulltime game developer responsible for a modestly successful game with full access to the industry in the US"

Emma Maassen

Grants and publishing deals aren't the only way Steam Spy numbers have been leveraged by small developers. Emma 'Eniko' Maassen was able to use the numbers to prove her business investment was legitimate when applying for a visa to move her studio, Kitsune Games, from the Netherlands to the US.

"I took a gamble with the game I was making [MidBoss] and my life's savings and applied for E2 status. In order to be eligible for E2 status you have to prove that your investment isn't marginal and that you're running a bona fide business. Steam Spy was very important in proving that, at least in part.

"We went through all the indie turn-based roguelikes on Steam and recorded the number of owners from Steam Spy as well as the percentage of positive reviews. We analyzed this data...to find a correlation between the percentage of positive reviews and number of owners using a scatter plot. We compiled this research and our conclusion for MidBoss' projected performance in a document for US immigration.

"E2 investor visas are particularly tricky to get, and around the time our application was approved we heard a few other E2 applications our attorney had worked on that had been denied. While the market research was only a part of a (very thick) file sent to USCIS, I think without the hard numbers from Steam Spy to substantiate our claims about MidBoss' performance, the E2 wouldn't have been approved and I wouldn't be where I am today; a fulltime game developer responsible for a modestly successful game with full access to the industry in the US."

The stories I heard not only ranged across different funding types, but also across different audiences receiving the pitches. Samuel Cohen of Altered Matter found Steam Spy data especially helpful as his team pitched its game, Etherborn, to a committee that lacked experience understanding the games industry.

"In 2015, after one year of development, we got selected for a video games incubator and accelerator program called GameBCN," he said. "As a result, we were presented with the opportunity to receive our initial funds from a local public entity and a bank, both of which had made an alliance for this program. But for that, we had to pitch the game in front of a committee who didn't necessarily know much about games, so the presented data needed to be particularly understandable. We can't afford the kind of reports that big companies use, so we had to ask some friend devs and document ourselves a lot with the available resources, Steam Spy being one of them."

"A game isn't good just because Hotline Miami sold a lot of copies. Tell us every single reason why Hotline Miami matters to your game"

John Cooney

"When it comes to talking with video game investors and publishers all this work is useful too, but in a different way. They already know their business. In such cases, showing our reports was more useful for them to see that we actually understood the numbers we were using."

That reinforcement from Cohen was a notion that I found to be a common thread among everyone I talked to: exact, specific numbers didn't matter. One of the criticisms Valve has leveraged against Steam Spy is that its data is inaccurate, and its creator has publicly agreed. But ballpark ideas of sales and users are still helpful not just for funding applications, but in the creation of the game itself. That's something Emmy Förster of Hastily Assembled Games found when working on Tracy Laser.

"We used Steam Spy data in order to get a feel for the overall market situation in VR," Förster said. "Rather than look at individual VR games, we were interested in general trends related to the particular game we were making. We both have a background in scientific data visualization and statistics, so the raw data provided by Steam Spy allowed us to extract a lot of information that we just weren't able to get out of ready-made data reports.

"In the end we were able to determine a target price point, play area size and refined the exact genre we were going for. While Steam Spy's data sets are, of course, estimates, these estimates nonetheless allowed us to back up a lot of our game design goals with actual data. We are certain that this played an integral part in us eventually receiving the funding."

Cohen and Förster may very well be right in their hypothesis that understanding the data made their pitches successful. John Cooney, senior manager of business development at Kongregate, emphasized the importance of grounded expectations for success relative to ballpark ideas of what other, similar games have done. For that, Steam Spy is important, but it's not the entirety of the picture.

"We get a lot of pitch decks where developers go for a super popular indie game in their genre or tag with a high owners number and paste it into their presentation, and explain to us that this a valid reason their game will be successful," Cooney said. "We end up seeing a lot of Nuclear Throne, The Binding of Isaac, and Hotline Miami because of that.

"We end up having to ask questions like, 'We think your game is really good, but do you think you'll sell as many as Hotline Miami?' Every once in a while the answer is yes, but more often it's there just as a big, flashy number to get investors to bite. A game isn't good just because Hotline Miami sold a lot of copies. Tell us every single reason why Hotline Miami matters to your game.

"We've been able to convince [Valve] on a few things, and we hope to convince them on this too."

Lars Doucet

"We want developers to unpack their comparable as much as possible. As a publisher, we ultimately value pitches where developers are able to conclude their own reasonable expectations on game performance using Steam Spy (or any other data point) as just a piece of the evidence. We're keen to know we're thinking the same way as the developer on vision, sales expectations, marketing positioning, and execution. If we're not thinking the same way about the game, it's going to be a hard relationship from the start.

"We also really want developers to value SteamSpy numbers the way we do, in that ballpark figures are exactly that. We want to know developers are being sensitive to their data sources and being thoughtful of how they use it."

That sensitivity and thoughtfulness can't exist at all, though, if there isn't at least a ballpark number to be sensitive or thoughtful about. For now, Steam Spy is soldiering on with its even-less-accurate picture of sales - pending further interference - though Valve recently made a direct effort to stop the service from improving at all when it shut down a new, more accurate method found by developer Tyler Glaiel to estimate Steam sales.

It seems, then, that Steam Spy will be unable to return to its former imperfect glory, though Doucet remains optimistic about developers' ability to convince Valve of the need for numbers.

"I'm not looking to be overly hostile or antagonistic to Valve," Doucet said. "We have an open line to them and they do actually listen to us sometimes, and I like to preserve that relationship. We've been able to convince them on a few things, and we hope to convince them on this too."

Galyonkin, too, is hopeful that Valve's eventual solution will be to provide even more accurate numbers across a wider breadth of data.

"There are plenty of things in Steam Spy that could be improved with Valve's access to data, that's why I'm so excited about their solution," Galyonkin said. "You could see if a person bought a game on Steam and if they're playing differently from people that got the same game elsewhere. DLC stats would be useful as well as overall increased accuracy for all stats. Valve was very adamant about being unhappy with the previous levels of precision of Steam Spy, so I naturally hope their solution will provide way more accurate data."