In defence of data-driven design

The advent of always-online gaming devices is arguably the single biggest change to the fabric of how games are made and sold that's happened in decades. Devices which are always connected (or at least mostly connected) have enabled new business models, new ways of playing and interacting, new ways of buying and paying for software - and even new ways of developing games, thanks to the novel possibility of flowing information from the player base back to the developer, who for the first time can see what real customers are actually doing in their game.

While I believe all of these to have been positive and worthwhile developments, albeit not a path without its bumps, twists and turns, I recognise that some of them are more troublesome than others. New distribution systems threaten to put retail stores out of business. New business models leave some demographics utterly cold. The occasionally mindless insistence on multiplayer and social features permeating every game can result in some terrible development decisions. There are downsides, in other words, but in each case they're outweighed by the upsides - new kinds of games, new kinds of interactions, new opportunities for creative teams and huge new demographic groups who are receptive to games for the first time.

In all of this, the one area I foolishly thought was uncontroversial was the improvement to the game development process that's been enabled by the reporting back of game data to the creators. The old model involved finishing a game, putting it on a disc and tossing it out into the wild, never to be heard from again; a developer's notion of what worked and what didn't could be gleaned only anecdotally, from the musings of professional reviewers or the comments of peers and colleagues. Even with the thorough post-mortem procedures in place at many good studios, you'd start working on your next game with a sound grasp of what went wrong in the previous title from an internal production perspective, but only the barest notion of what went right or wrong with the game in terms of the experience of your customers. The new model, which gives developers who set up appropriate reporting systems the ability to see detailed statistics on player interactions, and in many cases allows them to support the game post-launch by tweaking, fixing and improving weak areas in the design, seems indisputably better.

"The old model involved finishing a game, putting it on a disc and tossing it out into the wild, never to be heard from again"

I now think I need a new poster for the space over my desk, bearing the important slogan "Never say 'indisputably' on the Internet" - or in an academic paper, or in fact, at all, if you can manage it. It's an invitation to dispute equivalent to mooning and sticking out your tongue (perhaps not simultaneously) at the mercurial gods of Internet argumentation. Sure enough, then, the past few weeks have seen a backlash forming against the motion of using "metrics" in game design, with articles and comments on this site and elsewhere taking issue with this approach to game development as being a cynical and soulless effort to employ psychological ploys that exploit and abuse player trust - the very antithesis of creativity and passion.

The problem with these arguments, from my perspective at least, is that they're talking about a way of using metrics which makes absolutely no sense to me, and ignoring the reality of how talented game designers actually use this data. The implication of these arguments is that a studio which extensively uses metrics to improve a live game has somehow abandoned game design in favour of a dry and mathematical approach which simply iterates upon various combinations of variables until it reaches an apogee of monetisation, an apex point at which the cogs of the game's sundry equational gears mesh together in perfect synchronicity and money spontaneously pours forth from the wallets of the legions of dead-eyed, ceaselessly clicking drones who make up the playerbase.

That dystopian vision is either a tempting one for a science fiction story or a deeply hackneyed one for a forum rant about Candy Crush Saga, but it's not a reflection of any kind of objective reality. Perhaps there are companies out there who do use data in this manner, but I suspect they're not very successful companies, since a turd will remain unpolishable no matter how many statistical regressions you run on its crinkly lack of shininess. That's the baseline of this debate, in fact - there's absolutely no amount of post-hoc spreadsheet wizardry that can make the slightest bit of difference to a game that has lacked the creative spark of wonderful, compelling design in the first place - and that self-same spark of creativity is also essential to making dry, semi-comprehensible metrics into a useful tool to improve a game.

"Metrics do not replace game design, they inform it"

Metrics do not replace game design, they inform it. It's worth emphasising - metrics don't replace game design. They can't. What metrics can achieve, and do achieve wonderfully, is to give talented, insightful, creative game designers a solid basis from which to work. Metrics data can tell a designer if a large number of players are getting stuck on certain parts of a game, or not using certain actions in the intended way, or failing to grasp a key strategy which the designers had intended; it can tell the creative team if certain game sub-systems are being ignored, or if certain points in the game are causing players to give up entirely, or, yes, if certain transactions (be they for virtual or real currency) are unbalanced and not performing. What metrics data cannot do, however, is tell a designer how to fix any of these problems.

Metrics can tell you where a problem lies - where previously you might have known that most players don't finish your game, now you can know exactly where players give up and what approaches they've unsuccessfully tried to get past the stumbling block, for example. How you fix that problem - what insight you glean from your understanding of it (is it a level design issue, or indicative of a more fundamental game balance problem? Is the issue with the enemies, the itemisation, the physics?) and what action you take to fix it (does the boss just need less hitpoints, or can you keep the challenge the same while making the path to victory more clear?) are still firmly in the domain of the creative, experienced game designer. The spreadsheet can't provide any answers - all it can do is tell you what the right question is, which is a damn sight better than groping around in the dark and hoping you hit upon the right problems to work on.

Metrics can also, of course, tell you if your solution worked - this is how iterative testing functions. Implement a solution, test it and see if it worked; if not, think of something new. You can apply lots of clever methodological terms to that (throw Bayesian Learning into your vocabulary list for your next party appearance, it goes down a treat) but ultimately, the methodology associated with gathering and processing metrics, then running tests on various new cases you implement and iterating until something works, is a relatively cold mechanical process - yet one that is utterly meaningless without the old-school game design genius that's required in order to figure out what the data is telling you, come up with a creative response with it, and design the new systems to be tested at each iteration. Ignoring this side of the job - the most important part! - and claiming that designers have now been reduced to plodding spreadsheet-manipulators is a complete misrepresentation (and also rather ignores just how much of classic game design, for all its creative glory, has always at its best involved lots of plodding around spreadsheets).

"The ability to gather metrics from live games is a powerful and important tool, but its critics seem carried away in an over-estimation of that power and importance"

The ability to gather metrics from live games is a powerful and important tool, but its critics seem carried away in an over-estimation of that power and importance. Metrics is just data - utter gobbledegook without the insight and analysis of a great designer, and completely worthless except as a foundation for creative people to come up with clever solutions to the problems it describes.

I have no doubt that just as the critics of metrics overestimate its influence, so too do some of its most ardent supporters attempt to overuse data, failing to acknowledge and prioritise the continuing need for creativity, insight and inspiration in game design - perhaps even committing the cardinal sin of believing that it's possible to iteratively A/B test your way from scribbles to Shakespeare, which it most certainly is not. Those mistakes, though, are not representative of the wider picture of metrics and game design. Criticising the fantastic potential of data feedback and analysis in game development on the basis a few mistakes (and the slightly more subtextual basis of not liking micro-transactions very much, and assuming that the two are fundamentally linked) risks leaving a rather unhappy and extremely surprised baby blinking in the gutter, surrounded by rapidly cooling bathwater. The backlash is unfair; metrics is a great development in design, an essential skill for professional developers and a promising step for the future of this medium.