Close
Are you sure? Are you sure you want to report this comment? I understand, report it. Cancel

Devs can't "create magic" with a spreadsheet in front of them

Devs can't "create magic" with a spreadsheet in front of them

Mon 11 Nov 2013 5:03pm GMT / 12:03pm EST / 9:03am PST
MobileFree-to-Play

Sociologist Jen Whitson examines how analytics are shaping the industry for better and, more pressingly, for worse

It's well established by now that the use of metrics and advanced analytics can have a profound impact on a game's business. But it's less examined what kind of impact it has on a game's developers. Sociologist Jen Whitson, a researcher at Technoculture, Arts and Games, addressed the issue in a presentation today at the Montreal International Game Summit.

Whitson prefaced her talk by stressing that she wasn't going to advocate abandoning metrics entirely. It's a tremendous gift to know how the games we make impact their audiences, but there are significant dangers to it. Whitson is a sociologist who left the surveillance studies field to examine the game industry, but noticed some of the big data trends from her previous field creeping into her new one.

She began by talking about the Ho Chi Minh trail in 1968, years into the Vietnam War. The trail was a key supply route, and the US sought to break those supply lines in an interesting way. It air-dropped camoflauged sensors along the trail that tracked everything from trucks roaring along the trail to people peeing on the roadside. When the sensors picked these up, a napalm strike would be dropped into the jungle corridors to take the trucks out. However, the Vietnamese figured out what the Americans were doing and began intentionally confusing the sensors with recordings of truck sounds, and even launching water balloons filled with urine into desolate areas. The Americans dropped untold bombs based on their sensors, devastating jungles, accomplishing nothing of strategic import, and wasting incredible amounts of money.

Whitson said the lessons can be applied to social and mobile games, where data-driven design has become increasingly important.

"In many ways, I see devs as canaries in a coal mine," Whitson said.

The difficulties the industry is facing today, the studios shutting down and difficulties finding work, are not unique to gaming. They are instead another example of the evolution of capitalism. Over time, society has transitioned from family-run business to giant companies using assembly line work. While the benefits of the large company were clear, the work was numbing after a while, and people didn't enjoy a life spent as a drone doing one thing over and over. That has since evolved to a leaner process, where workers move from company to company and project to project, learning a multitude of skills and challenging themselves at every step. This stage, which Whitson said should be familiar in the industry, is a meritocracy, but one that makes job security a thing of the past, where developers need to constantly update their skillset.

The gaming industry is undergoing a massive change after decades of console dominance, Whitson said, an era characterized by inequal power relationships between manufacturers, publishers, and players.

"Game development is the very epitome of precarious labor," Whitson said.

In any given week, a quarter of developers will work more than 50 hours a week, Whitson said. But typical weeks aren't always so typical, with crunch lasting as long as a year or more, and some studios operating in perpetual crunch. At the same time, profit margins in the console world are constantly going down, with only one out of every 25 games that starts production ever being profitable. The whole AAA scenario is similar to working the assembly line on the factory floor, Whitson said.

But things are changing. Whitson pointed to Angry Birds as an example. The game was made originally for a couple hundred thousand dollars, and has since ballooned into a half-billion dollar empire. It's an encouraging counterpoint to the AAA market, but big problems remain.

"Rovio made 51 games and almost went bankrupt before winning the lottery with Angry Birds," Whitson said.

The way such a hit could be made was with metrics, Whitson said. Through easily trackable metrics and A-B testing, it was easy for casual game developers to use real-time feedback to maximize revenue around whatever happened to work best. The results can be difficult to argue with, especially in the absence of other explanations for a game's success.

"The game industry can't understand why s***** games like Famrville came out of nowhere to make money," Whitson said. "It couldn't be the game itself, so it had to have something to do with the data under the hood."

The big change of metrics is that the design of the game is no longer dictated from above, Whitson said. Now, it's effectively "crowdsourced," determined by the users. One problem with this is that it drives success to the local maxima, the highest peak in the immediate vicinity. If there is a larger peak that can be reached, but only by going through a valley, reliance on metrics will keep games from reaching it.

"Metrics are really, really seductive, especially when they're seen as a lifeline for the industry," Whitson said.

There is also a belief that players don't know what they want, and aren't to be trusted over the data. When Battlefield Heroes suggested making powerful weapons available in the game, EA found that those who were the most vocal against the change were some of the biggest spenders on the new weapons and revenues doubled almost overnight. So while the casual and mobile revolution promised a change from the assembly line development of the AAA market, it has actually eliminated creative autonomy. Instead of being controlled by assembly line managers, developers now effectively have robot overlords in the form of metrics.

Metrics, by their nature, stifle innovation, Whitson said. They are excellent for optimization, but terrible for innovation. If a studio is all live operations and metrics-driven, the developers on the team will be hungry for more creative outlets. If a studio doesn't offer that, the talent will go elsewhere.

Secondly, Whitson said once developers get in a metrics-driven mindset, one where minor changes are made with clear evidence to say what was right or wrong, they are limited from considering other ways of thinking. They stop considering whether the game's secret sauce is its sense of humor, the real-world experience shared by players enjoying the game with one another. It's difficult, if not impossible, to hold both of those mindsets at the same time, like patting your head and rubbing your belly at the same time.

"If you want your devs to create magic, you can't also put the spreadsheet in front of them," Whitson said.

Third, stats can lie. They can tell you what your players are doing, but not why. And that makes them tell only half the story. On top of that, they can be massaged to make whatever point is wished. And since numbers are convincing, whichever person in a design argument has the numbers is likely to win. Basing something on numbers instead of intuition seems more rational. However, those who hate metrics the most should familiarize themselves with them so as to better make the case for their own gut instincts, Whitson said.

Fourth, analytics must be tempered by ethics, Whitson said. Developers need to talk openly within their companies about where they will draw the line. The decisions to cap spending or hunt for whales has repercussions for studios, possibly leading to layoffs and other such measures, but they should be made together.

Finally, metrics are a symptom of managing a risky industry, Whitson said. They can be escaped by running a Kickstarter campaign or focusing on niche markets, but it's risky and unpredictable. Whitson knows that metrics aren't going anywhere, but she stressed the need for people to consider them as just one tool in a set.

"We need to focus on making our industry a place to live in rather than a place to pass through before we burn out," Whitson said.

10 Comments

Jim Webb
Executive Editor/Community Director

2,266 2,405 1.1
Popular Comment
I prefer to think of it simply as the greatest artists in history never painted by numbers.

Posted:10 months ago

#1

Tom Keresztes
Programmer

661 271 0.4
[who]

Edited 2 times. Last edit by Tom Keresztes on 12th November 2013 8:57am

Posted:10 months ago

#2
I agree, If all there was to making a great game was to throw any game idea up against a wall, then just test/tweak and iterate ad nauseam, it would be simple to make great games and money. Unfortunately as with anything with art it doesnt work that way. Spreadsheets and so forth offer good CYA after the smoke clears, but as far as making something truly special, spreadsheets etc are not worth much.

Now if you just want to make the man a little money, then by all means follow the spreadsheet, make clone prequel sequel number 5 and have at it. But truth be told at that point you might as well just work at a sausage factory. Your input and ideas are not welcome, just follow the recipe.

Edited 3 times. Last edit by Todd Weidner on 12th November 2013 4:48pm

Posted:10 months ago

#3

Jeffrey Kesselman
Professor - Game Development

30 52 1.7
In the end, metrics are a part of marketing and, to quote one of the smartest industry execs i have ever known, "The problem with marketing is it can only tell you what the last hit was, not what the next hit will be."

What metrics are good for is driving player behavior. This is why the micro-transaction market fell so deeply in love with them. Design of micro-transaction based games is all about cynically driving the player to a buying behavior.

What metrics are BAD at is saying what the subjective player experience is. And for those of us who got into this business not to try to rake out a buck but to try to excite people and make them happy, that makes them almost useless.

I say almost because the OTHER thing they are good for are questions that really are purely numerical, like play balance.

Edited 1 times. Last edit by Jeffrey Kesselman on 12th November 2013 4:55pm

Posted:10 months ago

#4

Brian Lewis
Operations Manager

132 84 0.6
You keep using that word.... metrics. I do not think it means what you think it means.

Metrics are a great way to measure the results of something. However, they dont drive the actual behavior. The first step of a metric driven approach is to try new, and innovative things... so that you can find out which actually work better. Good metrics can only exist in an creative and innovative enviournment. If people are unwilling/unable to try new things, then they dont need metrics to tell them what worked.... because it will not matter.

Posted:10 months ago

#5

Paul Johnson
Managing Director / Lead code monkey

854 1,112 1.3
Most of my last games have been all about the spreadsheets. Try making a TBS or CCG without one... :)

Posted:10 months ago

#6
Paul, you're right. I was waiting for someone to bring that up in response to the title.
Lots of the most dedicated playing experiences are also about spreadsheets - like theorycrafting.
In this case, I was using the word spreadsheets to stand-in for a focus on monetization features like engagement and retention.
Not using analytics for things like game balancing would be incredibly short-sighted.

Posted:10 months ago

#7
Brian, I agree that fully that good metrics exist in creative environments. This is what Tayber Voyer was arguing in his talk (also covered by GamesIndustry.biz).
However, I don't agree with the rest of your statement. There is a hill-climbing problem introduced by metrics - they're great for making incremental improvements and measuring their effect. In this usage, they don't tell you WHY players responded positively (that's why Tayber Voyer recommends coupling metrics with forum feedback and community engagement), and they are more effective when you are only changing one small variable (i.e. the loot drop size, or a banner color).
Metrics are like running a lab experiment. You only change one small thing at a time, otherwise you end up with muddied results. Larger innovations mean that you change many variables at once, thus it becomes very difficult to pin down just which variable was responsible for your results. You can clear up some of this muddiness by then going to players and digging deeper to try and pin down what caused the improvement. But I suspect many studios don't have the resources or the time to do this.

Edited 1 times. Last edit by Jen Whitson on 13th November 2013 3:16pm

Posted:10 months ago

#8

Richard Vaught
Studying B.A. in Game Design

19 33 1.7
Are you saying Zynga was "creative and innovative"? Or any of the other FB/Android/iOS metrics driven clones? Are you suggesting that they DIDN'T use metrics to drive player behavior? ... ... ...

Posted:10 months ago

#9

Paul Johnson
Managing Director / Lead code monkey

854 1,112 1.3
Actually we don't bother with any sort of analytics. If we noticed that 50% of players left the game after tutorial #3 for example, and we check out tutorial #3 and it seems interesting, then what?

From that, all we can learn is that people who don't like our game give it to about tutorial #3 before uninstalling. How is that better than a general non-analytics version; "people who don't like our game won't play it"

There is a whole industry springing up around these measuring systems, but I think all but the top publishers with daily plays in the millions can't make any meaningful use out of it. And only then for measuring tiny changes over long periods.

Posted:10 months ago

#10

Login or register to post

Take part in the GamesIndustry community

Register now