Devs must be careful with "massaged" metrics

Execution Labs' Jen Whitson details dangers of data-driven design and why it's here to stay anyway

Last month at the Montreal International Game Summit, PhD researcher Jen Whitson spoke about the negative impacts data-driven design can have on developers. The message was pulled from years of research as well as her own experience as an "embedded ethnographer" with start-up incubator Execution Labs, but as she told GamesIndustry International after her talk, it's not always one developers are eager to hear.

"Of course there has to be some sort of defensive quality," Whitson said of developers' reactions to her work. "You're saying that how you make your money has negative consequences. Obviously, we don't want to hear that in the industry. So pushback is natural, and I don't expect everybody to go out and drastically change how they work."

Instead, Whitson said her goal with the talk was to provoke discussion, to open up a list of possibilities for people to think about their workplaces in different ways. In some ways, the issue parallels that of crunch in the development world, and progress in addressing either could be slowed by people who equate them with financial gain. There's no shortage of successful games produced by teams that crunched, just as it's not hard to find wildly successful games with a reliance on data-driven design.

"We don't want to face some of these ugly things about our industry head on because the way we face them may mean that we're putting other people out of work, which is hideous to think about," Whitson said.

"My fear is that metrics are used to turn people into data just because they're much more tractable than actually dealing with real humans."

Jen Whitson

The problem, as she explained in her talk, is that developers are using metrics not because they're the best tool for the job, but because they're a substitute for player feedback that can be difficult, time-consuming, or even ego-bruising to collect and sift through.

"My fear is that metrics are used to turn people into data just because they're much more tractable than actually dealing with real humans," Whitson said.

There's also a danger that advocates of analytics put too much faith in the numbers, not just in guiding their design decisions, but in trusting where they came from.

"It's important to educate ourselves and to know how the metrics and analytics systems work, and to pay attention to how the messiness of actual player behavior is cleaned up for the data and then transported back to us," Whitson said. "We're taking that 'raw data' as the truth, but in some sense it's cleaned up and massaged along the way. It's almost like a telephone game."

Whitson isn't the only one with misgivings about data-driven design. Her impression on the situation is that it's not going anywhere, even though the "vast majority" of developers (including those who use it in their own games) dislike it. But given how sharply the industry has turned toward the practice, familiarizing oneself with it isn't just a design concern, it's job security.

"There's an idea that metrics and analytics technology is progressing so quickly, and there's so many third-party service vendors and different hooks you need to add in that if you don't get in on the ground floor and learn and test and know how to do these things now, the industry may move on without you," Whitson said. "You won't have picked up these skills about dealing with metrics and dealing with big data in the industry. And then you'll be truly f*****."

Latest comments (6)

Christophe Danguien games developer 4 years ago
She has to be careful when talking data-driven. I bet I am not the only one that understand data-driven by the behaviour of the game change with the assets providing.
I think she means people-behaviour-driven here...or is it me understanding wrong ?
0Sign inorRegisterto rate and reply
Justin Mingo Marketing 4 years ago
I think what she's saying is that like any tool or asset: it has to be taken with a grain of salt, and not used like a "magic bullet". The use of data like this can help a lot, however it has weaknesses, and as such: it should be used in conjunction with other research and feedback methods. It should not replace your other development research.
0Sign inorRegisterto rate and reply
Tom Keresztes Programmer 4 years ago
I think she means people-behaviour-driven here...or is it me understanding wrong ?
She is speaking the marketing lingo, not the engineering one.
0Sign inorRegisterto rate and reply
Show all comments (6)
Tim Carter Designer - Writer - Producer 4 years ago
We don't need "years of research" to see when scientists have taken over in an art medium... and just how messed up that is.

I can tell you with 5 seconds of reseach that having white labcoats running creative is a mistake.

Science is only one means of understanding our world. Art is different, but just as valid.

There is nothing new to this epistemological argument. We've been here before. It's typical scientific hubris.

Edited 5 times. Last edit by Tim Carter on 5th December 2013 7:12pm

1Sign inorRegisterto rate and reply
Keldon Alleyne Developer, leader, writer, Avasopht Ltd4 years ago
People who lack an artistic aptitude can only connect with the form via metrics. They can't hear an arrangement of musical notes and assess its intrinsic beauty, instead they must consult the opinions of others.

But the successful creators have an intuition of how people will respond and feel because they have a true understanding of the human connection to the experience of a medium.
2Sign inorRegisterto rate and reply
David Serrano Freelancer 4 years ago
Metrics can and should be used to remove guess work from creative and technical process. Because if developers can clearly defined specific parameters and tolerances within a target audience, they can use the data to make the informed decisions needed to increase the likelihood of a game eliciting the desired response from the audience.

So those who believe metrics are a replacement for talent or skill don't understand the purpose or benefit of collecting and using the data. Metrics can't replace talent or skill, but they can enhance it and allow you to point it in the most productive or efficient direction. And in my experience, systems and procedures designed to remove guess work from a creative or technical process will frequently expose people who use the pretense of expertise or subjectivity to conceal a lack of creative or technical talent, skill and knowledge.
0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.