"Retention is not the same as happiness"

Free Realms creative lead Laralyn McWilliams describes the pitfalls of metrics-driven game design

Laralyn McWilliams, the former creative lead on SOE's Free Realms, has warned free-to-play developers away from using metrics as the basis for the roadmap of their games. Data can indicate a great deal about a game, she argued, but it falls short in the area that matters most.

Relative to the growth of the free-to-play business model outside of Asian markets, McWilliams is something of an old hand. As creative lead on Free Realms, she was instrumental in the creation of what was, at that time, one of the most ambitious free-to-play games ever attempted in the West.

However, after nearly a decade of thinking about and working with the free-to-play model, she has reached certain conclusions about the dominance of metrics in the way games are designed. Fundamentally, metrics cannot indicate the player's emotional attachment to the game, and in some cases they can be misleading.

"There's no measuring spoon for love. You can't quantify it," she said in a session at Casual Connect Europe in Amsterdam. "Retention is not the same as happiness."

In all media, the consumer often continues to engage long after their satisfaction with the experience has started to wane. People watch a TV show even when they're bored with its progress, they read books in a series that has long since peaked, and they play games even when they have started to lose interest in - and even resent - the experience.

For McWilliams, metrics have failed to offer a clear picture of when this process starts to happen, and the motivations that set it in motion. She used A/B tests as an example of a method that seems to be effective, but is often misleading due to the complex and dynamic nature of online games. A/B tests do not take place in a controlled environment, yet major decisions about a game's design are directly informed by their results.

"Really, after a few months in your game, every player is a Frankenstein of all the A/B tests and updates you've ever done," she said. "The decisions you make from an A/B test are not necessarily good decisions for the game as a whole."

According to McWilliams, the sort of granular detail metrics offer can be seductive, granting a sense of control and confidence. But the reality of operating a game in a live environment is akin to climbing a high rock-face. The path to the top is impossible to plan from the ground. You can only reach out and try to find the next handhold, and strong instincts can be just as useful as knowledge of the topography.

"The point is that It's a mix of logic and emotion that goes into our decisions as game designers," she said. "That's why we can never design by metrics."

McWilliams did not pretend to have the definitive answer to making the player happier. It is, she maintained, very much an open question, and one worthy of detailed discussion.

However, she was more decisive on one matter. Pulling up a slide of a chart depicting the average player lifespan on mobile platforms - descending rapidly with the passage of time - she said, "I maintain that this is crazy bullshit." She pointed to EverQuest as an example of a game that managed to build a lasting relationship with its players, and without the high churn rate that is basically expected by so many developers.

Speaking in the subsequent Q&A session, Teut Weidemann, senior online game superviser at Ubisoft, agreed wholeheartedly with McWilliams' stance.

"People forget that we're in the online games business," he said. "People see the metrics and only think about the monetisation. At Ubisoft we call players 'fans'. It makes you think differently."

More stories

The coming recession will hit the industry hard | Opinion

It's a long time since the games business was called 'recession-proof' - and recent changes could have made it more vulnerable to economic downturns than ever before

By Rob Fahey

Brazilian publisher DX Gameworks sets up shop

Outfit will focus on console and PC, offers variety of services including funding, porting, localization

By Brendan Sinclair

Latest comments (13)

shhh, the bean counters will not approve of this truism :)
4Sign inorRegisterto rate and reply
Brian Lewis Operations Manager, PlayNext8 years ago
As we all know, '90% of all statistics are made up'. In the same vein, the meaning of most metrics are made up as well. Metric driven anything is only as good as the metrics (and the interpretation of them). This is true in any business, and often is the result of the use of stats/metrics to prove a point, rather than using them to determine the point.

If you have a preconceived notion, and only look for proof of that notion, then it is easy to find the supporting data (and ignore anything that doesn’t fit the notion). If, instead, you look at a wide range of data, and use it to determine a result (that includes all the data) it is more likely to be accurate. The same applies to metrics. If (for example) the assumption is that DAU is the key indicator of a healthy game, it is easy to generate numbers to prove that… up until the game dies. The same applies to all design choices, and to most metrics used by the industry. They’re only as good as they are valid… which is why they should always be questioned.
2Sign inorRegisterto rate and reply
Murray Lorden Game Designer & Developer, MUZBOZ8 years ago
Great article, it's lovely to see someone with so much experience in the area raising some counter-arguments against metric driven design.

Of course - as long as metric driven design yields good profits - it'll remain popular. And it obviously has some useful purposes in practice.

But I'd hope that general game design intuition - with the goal of producing an engaging, fun, enriching, rewarding game experience, can remain near the forefront too!
0Sign inorRegisterto rate and reply
Show all comments (13)
Nicholas Lovell Founder, Gamesbrief8 years ago
I'm often thought of as a bean counter. I approve wholeheartedly.
0Sign inorRegisterto rate and reply
Nicholas Lovell Founder, Gamesbrief8 years ago
Metrics are a tool. They are an incredibly useful tool that can help you identify where you have issues in a game, and to help you understand whether the thing you did to improve the game helped or made it worse. They are as useful as a designer's instinct and more useful than listening to the vocal minority in your forums or Twitter to help you make better games.

As always, though, the metrics are a target. A target is a proxy for an objective. If you shoot for the target, instead of the objective, you will get in all kinds of trouble.
2Sign inorRegisterto rate and reply
Paul Johnson Managing Director / Lead code monkey, Rubicon Development8 years ago
I got blasted by the experts for a similar sentiment, but I still agree with it.

Metrics *can* give some useful numbers, but it's measuring secondary details just because the primary details aren't measurable. I think it's much better to engage with consumers as much as possible and try to get a feel for what should be improved that way.

Because once you have enough evidence of the problem, it really will be the actual problem and you can fix it with certainty. I have direct experience of this.

Edited 2 times. Last edit by Paul Johnson on 13th February 2014 9:11am

1Sign inorRegisterto rate and reply
Paul Johnson Managing Director / Lead code monkey, Rubicon Development8 years ago
There's also an element of doing it "by the numbers". Games are meant to be an expression of what the developer feels is a good game, which includes making their own decisions.

Using metrics can indicate suggested changes, but when you stop thinking for yourself you lose a bit of soul. If you *only* want to make money, there are safer investment industries you can to move to.
1Sign inorRegisterto rate and reply
Claas Grimm CRO, Red Hot CG8 years ago
"Love can't buy you money" - Motorhead, 1996
1Sign inorRegisterto rate and reply
Alfonso Sexto Lead Tester, Ubisoft Germany8 years ago
Seeing how online rage runs havoc in the most elitist sector of "World of Warcraft" and "DOTA2" I wonder if that people actually is having fun,

I think "addiction" is the word in those cases.
0Sign inorRegisterto rate and reply
Christophe Danguien games developer 8 years ago
Did some people really need an article to know that....really ?
1Sign inorRegisterto rate and reply
David Serrano Freelancer 8 years ago
It's a mix of logic and emotion that goes into our decisions as game designers, she said. That's why we can never design by metrics.
On one hand, she's right. Metrics will never be a substitute or replacement for creative talent. But on the other hand, this is exactly why Chris Bateman wrote in 21st Century Game Design: "What is clear is that designers who design games solely to please their own sensibilities are unlikely to create genuinely mass market games."
0Sign inorRegisterto rate and reply
Gareth Eckley Commercial Analyst 8 years ago
I don't really like this bashing of games metrics. It's all very trendy, but the problems with metric assisted design are in the implementation. The underlying usage of maths to help understand the behaviour of systems is pretty well founded by now.

Metrics can only measure the things the developers want them to measure. An excellent example of how they can be subverted is with the recent Dungeon Keeper approval rating mechanism.

If you want to measure "love of a game" then you need to first analyse and then define the behaviours that are typical to highly invested players. This can only be done by a mixture of analytical and creative thinking and is where marketing and analytics can add real value to a product by assessing real player behaviour and tying that to in game event paths and numbers. This behaviour will always be game specific and require total honesty on the part of all the stakeholders.

In the above scenario, metrics can give extremely useful insights into game behaviour - or which player/design choices ultimately end up in player dissatisfaction. However, much like a parliamentary democracy, there's a lot of scope for the "facts" to be either biased to begin with, measure unimportant values or simply be ignored or re-interpreted as desired.

Assume that 10% of players have player housing. It is not the fault of the analytics team when someone in the creative side of the team decide that this means that 90% of players enjoy the fresh air more.

As Brian Lewis mentions above,

"If you have a preconceived notion, and only look for proof of that notion, then it is easy to find the supporting data..."

This is a highly prevalent attitude amongst people using metrics. They merely use it as another method to further their agenda.
0Sign inorRegisterto rate and reply
Laralyn McWilliams Studio Design Director, The Workshop Entertainment8 years ago
There was no bashing of metrics in my talk, and I don't interpret the article that way either. I was one of the first designers to talk at conferences about integrating metrics into the design process, in fact, and I've been a proponent of them for years!

The focus of the talk is on the importance of balance. I've personally experienced decisions about game design being made based on metrics alone, and my point is that metrics only reflect behavior, which is half the picture. Emotion is the other half, and metrics can only vaguely hint at player emotion so we need a more holistic way to solve problems that incorporates both aspects.

On the other hand, you say: "If you want to measure "love of a game" then you need to first analyse and then define the behaviours that are typical to highly invested players."

Yes, but a key aspect of my talk is that invested behavior is NOT the same as happiness, and it is not automatically a measurement of "love of a game." It can be, for some players, but it's a dangerous assumption to paint every player with that brush stroke. Consumers continue with an investment past the point where they were genuinely happy--hoping it gets better, because it's a habit, because a friend is pressuring them, or other reasons. If your process assumes that just because he's still playing he's happy, you'll completely miss the fact that he's already begun the mental process of churn and by the time you realize it he may have turned a corner past redemption by any offers or win-back campaign.
0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.