In defence of data-driven design

Data can never replace design genius, only complement it

The advent of always-online gaming devices is arguably the single biggest change to the fabric of how games are made and sold that's happened in decades. Devices which are always connected (or at least mostly connected) have enabled new business models, new ways of playing and interacting, new ways of buying and paying for software - and even new ways of developing games, thanks to the novel possibility of flowing information from the player base back to the developer, who for the first time can see what real customers are actually doing in their game.

While I believe all of these to have been positive and worthwhile developments, albeit not a path without its bumps, twists and turns, I recognise that some of them are more troublesome than others. New distribution systems threaten to put retail stores out of business. New business models leave some demographics utterly cold. The occasionally mindless insistence on multiplayer and social features permeating every game can result in some terrible development decisions. There are downsides, in other words, but in each case they're outweighed by the upsides - new kinds of games, new kinds of interactions, new opportunities for creative teams and huge new demographic groups who are receptive to games for the first time.

In all of this, the one area I foolishly thought was uncontroversial was the improvement to the game development process that's been enabled by the reporting back of game data to the creators. The old model involved finishing a game, putting it on a disc and tossing it out into the wild, never to be heard from again; a developer's notion of what worked and what didn't could be gleaned only anecdotally, from the musings of professional reviewers or the comments of peers and colleagues. Even with the thorough post-mortem procedures in place at many good studios, you'd start working on your next game with a sound grasp of what went wrong in the previous title from an internal production perspective, but only the barest notion of what went right or wrong with the game in terms of the experience of your customers. The new model, which gives developers who set up appropriate reporting systems the ability to see detailed statistics on player interactions, and in many cases allows them to support the game post-launch by tweaking, fixing and improving weak areas in the design, seems indisputably better.

"The old model involved finishing a game, putting it on a disc and tossing it out into the wild, never to be heard from again"

I now think I need a new poster for the space over my desk, bearing the important slogan "Never say 'indisputably' on the Internet" - or in an academic paper, or in fact, at all, if you can manage it. It's an invitation to dispute equivalent to mooning and sticking out your tongue (perhaps not simultaneously) at the mercurial gods of Internet argumentation. Sure enough, then, the past few weeks have seen a backlash forming against the motion of using "metrics" in game design, with articles and comments on this site and elsewhere taking issue with this approach to game development as being a cynical and soulless effort to employ psychological ploys that exploit and abuse player trust - the very antithesis of creativity and passion.

The problem with these arguments, from my perspective at least, is that they're talking about a way of using metrics which makes absolutely no sense to me, and ignoring the reality of how talented game designers actually use this data. The implication of these arguments is that a studio which extensively uses metrics to improve a live game has somehow abandoned game design in favour of a dry and mathematical approach which simply iterates upon various combinations of variables until it reaches an apogee of monetisation, an apex point at which the cogs of the game's sundry equational gears mesh together in perfect synchronicity and money spontaneously pours forth from the wallets of the legions of dead-eyed, ceaselessly clicking drones who make up the playerbase.

That dystopian vision is either a tempting one for a science fiction story or a deeply hackneyed one for a forum rant about Candy Crush Saga, but it's not a reflection of any kind of objective reality. Perhaps there are companies out there who do use data in this manner, but I suspect they're not very successful companies, since a turd will remain unpolishable no matter how many statistical regressions you run on its crinkly lack of shininess. That's the baseline of this debate, in fact - there's absolutely no amount of post-hoc spreadsheet wizardry that can make the slightest bit of difference to a game that has lacked the creative spark of wonderful, compelling design in the first place - and that self-same spark of creativity is also essential to making dry, semi-comprehensible metrics into a useful tool to improve a game.

"Metrics do not replace game design, they inform it"

Metrics do not replace game design, they inform it. It's worth emphasising - metrics don't replace game design. They can't. What metrics can achieve, and do achieve wonderfully, is to give talented, insightful, creative game designers a solid basis from which to work. Metrics data can tell a designer if a large number of players are getting stuck on certain parts of a game, or not using certain actions in the intended way, or failing to grasp a key strategy which the designers had intended; it can tell the creative team if certain game sub-systems are being ignored, or if certain points in the game are causing players to give up entirely, or, yes, if certain transactions (be they for virtual or real currency) are unbalanced and not performing. What metrics data cannot do, however, is tell a designer how to fix any of these problems.

Metrics can tell you where a problem lies - where previously you might have known that most players don't finish your game, now you can know exactly where players give up and what approaches they've unsuccessfully tried to get past the stumbling block, for example. How you fix that problem - what insight you glean from your understanding of it (is it a level design issue, or indicative of a more fundamental game balance problem? Is the issue with the enemies, the itemisation, the physics?) and what action you take to fix it (does the boss just need less hitpoints, or can you keep the challenge the same while making the path to victory more clear?) are still firmly in the domain of the creative, experienced game designer. The spreadsheet can't provide any answers - all it can do is tell you what the right question is, which is a damn sight better than groping around in the dark and hoping you hit upon the right problems to work on.

Metrics can also, of course, tell you if your solution worked - this is how iterative testing functions. Implement a solution, test it and see if it worked; if not, think of something new. You can apply lots of clever methodological terms to that (throw Bayesian Learning into your vocabulary list for your next party appearance, it goes down a treat) but ultimately, the methodology associated with gathering and processing metrics, then running tests on various new cases you implement and iterating until something works, is a relatively cold mechanical process - yet one that is utterly meaningless without the old-school game design genius that's required in order to figure out what the data is telling you, come up with a creative response with it, and design the new systems to be tested at each iteration. Ignoring this side of the job - the most important part! - and claiming that designers have now been reduced to plodding spreadsheet-manipulators is a complete misrepresentation (and also rather ignores just how much of classic game design, for all its creative glory, has always at its best involved lots of plodding around spreadsheets).

"The ability to gather metrics from live games is a powerful and important tool, but its critics seem carried away in an over-estimation of that power and importance"

The ability to gather metrics from live games is a powerful and important tool, but its critics seem carried away in an over-estimation of that power and importance. Metrics is just data - utter gobbledegook without the insight and analysis of a great designer, and completely worthless except as a foundation for creative people to come up with clever solutions to the problems it describes.

I have no doubt that just as the critics of metrics overestimate its influence, so too do some of its most ardent supporters attempt to overuse data, failing to acknowledge and prioritise the continuing need for creativity, insight and inspiration in game design - perhaps even committing the cardinal sin of believing that it's possible to iteratively A/B test your way from scribbles to Shakespeare, which it most certainly is not. Those mistakes, though, are not representative of the wider picture of metrics and game design. Criticising the fantastic potential of data feedback and analysis in game development on the basis a few mistakes (and the slightly more subtextual basis of not liking micro-transactions very much, and assuming that the two are fundamentally linked) risks leaving a rather unhappy and extremely surprised baby blinking in the gutter, surrounded by rapidly cooling bathwater. The backlash is unfair; metrics is a great development in design, an essential skill for professional developers and a promising step for the future of this medium.

Latest comments (8)

James Prendergast Research Chemist 4 years ago
The problem with these arguments, from my perspective at least, is that they're talking about a way of using metrics which makes absolutely no sense to me, and ignoring the reality of how talented game designers actually use this data.

Isn't that the point then? There are probably people that are misusing metrics and those articles and people want to get the message out before it's too late...

It's worth pointing out that industries and people do make mistakes and take mis-steps and without some discussion on those topics a "de facto" method of operation can become dyed into the business very easily and permeate it for years to come. Anything from the misguided "more hours worked equals an equal amount of increased output" to "outsourcing is always best" are good examples of this. As was said in one of the other articles: the person with the numbers to back up their assertion usually wins a discussion... the problem is that I can see that being a very easy hole to fall into without meaning to.

None of those other articles said that metrics were bad and shouldn't be utilised - only that they can be misused. That's basically the same premise as this article too...
2Sign inorRegisterto rate and reply
Renaud Charpentier Game Director, The Creative Assembly4 years ago
There are 2 fundamental limits to metrics:
1> They will only inform you about what is already in the game, never about what it lacks
2> They will tell you that when the core of the game is already finished

So metrics are very useful for balancing but they are irrelevant to what is actual game design: creating the core of the game.
8Sign inorRegisterto rate and reply
Dave Herod Senior Programmer, Codemasters4 years ago
Just one bit of inaccuracy, I'm afraid:
since a turd will remain unpolishable
You can indeed polish a turd
2Sign inorRegisterto rate and reply
Show all comments (8)
Scott Davis Games Analyst, Lift London, Microsoft4 years ago
@ Renaud
I think you may be rushing into those two statements a bit too harshly - with the right data (and right analysts), you can certainly see when the core loop of your game is broken and can help infer that something is indeed missing from your game.

The data will never tell you what exactly is missing from your game, no one should ever think that data driven design doesn't require any human input, but by analysing drop out points and identifying key breakages in the core of your game you can effectively work with designers to plug that gap.

Data should never replace the intuition of designers, it should be there to work with them....and it should also be there to remind them when their intuition had produced some bad ideas.

Edited 1 times. Last edit by Scott Davis on 15th November 2013 4:58pm

1Sign inorRegisterto rate and reply
Is this Metrics argument still going on? All the cool kids are arguing about f2p now.
1Sign inorRegisterto rate and reply
Tim Carter Designer - Writer - Producer 4 years ago
"Data can never replace design genius, only complement it"...


Tell that to the Suits.
1Sign inorRegisterto rate and reply
@Tim Amen.
0Sign inorRegisterto rate and reply
Renaud Charpentier Game Director, The Creative Assembly4 years ago
@ Davis

To get metrics on your core loop, well, you need the core loop in a state that is good enough so alpha players can actually go through it. When you have that, you are already in closed beta, whatever you call it, and it means 80%+ of your dev is done, including most of the art/ui/backend etc... So yes, at that stage it is useful, but sadly, from a production point of view it is already very late.
If you have "small" problems tweaking your systems can solve them, that's what I call balancing. If you have "big" problems, if it it tells you the core loop of your game is actually broken you are in deep troubles. You need to go back to gameplay prototyping and that project will likely be killed. Might still be better than wasting more money on releasing it, but as I said, that comes late.

Same song with the more traditional playtests by the way, so it's not a metric approach limit, it's the problem of evaluating any subjective creative process. Until a movie, a play, a book is finished, it's very difficult to evaluate where you are... and before you can confront any creation to an audience, most of the risk/cost/time have already been taken. Video Games are no exception.

That's a personal view but I tend to see metrics as a tool to help turn a good game into a great one, through correct balancing and useful updates. I don't think it can save a weak game or every iOS release these days would polish its way to the top. 99% are now monitored through metrics, 99.9% never make it to the top.
0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.