Close
Report Comment to a Moderator Our Moderators review all comments for abusive and offensive language, and ensure comments are from Verified Users only.
Please report a comment only if you feel it requires our urgent attention.
I understand, report it. Cancel

"Retention is not the same as happiness"

"Retention is not the same as happiness"

Wed 12 Feb 2014 10:56am GMT / 5:56am EST / 2:56am PST
PublishingDevelopmentCasual Connect

Free Realms creative lead Laralyn McWilliams describes the pitfalls of metrics-driven game design

Laralyn McWilliams, the former creative lead on SOE's Free Realms, has warned free-to-play developers away from using metrics as the basis for the roadmap of their games. Data can indicate a great deal about a game, she argued, but it falls short in the area that matters most.

Relative to the growth of the free-to-play business model outside of Asian markets, McWilliams is something of an old hand. As creative lead on Free Realms, she was instrumental in the creation of what was, at that time, one of the most ambitious free-to-play games ever attempted in the West.

However, after nearly a decade of thinking about and working with the free-to-play model, she has reached certain conclusions about the dominance of metrics in the way games are designed. Fundamentally, metrics cannot indicate the player's emotional attachment to the game, and in some cases they can be misleading.

"There's no measuring spoon for love. You can't quantify it," she said in a session at Casual Connect Europe in Amsterdam. "Retention is not the same as happiness."

In all media, the consumer often continues to engage long after their satisfaction with the experience has started to wane. People watch a TV show even when they're bored with its progress, they read books in a series that has long since peaked, and they play games even when they have started to lose interest in - and even resent - the experience.

For McWilliams, metrics have failed to offer a clear picture of when this process starts to happen, and the motivations that set it in motion. She used A/B tests as an example of a method that seems to be effective, but is often misleading due to the complex and dynamic nature of online games. A/B tests do not take place in a controlled environment, yet major decisions about a game's design are directly informed by their results.

"Really, after a few months in your game, every player is a Frankenstein of all the A/B tests and updates you've ever done," she said. "The decisions you make from an A/B test are not necessarily good decisions for the game as a whole."

According to McWilliams, the sort of granular detail metrics offer can be seductive, granting a sense of control and confidence. But the reality of operating a game in a live environment is akin to climbing a high rock-face. The path to the top is impossible to plan from the ground. You can only reach out and try to find the next handhold, and strong instincts can be just as useful as knowledge of the topography.

"The point is that It's a mix of logic and emotion that goes into our decisions as game designers," she said. "That's why we can never design by metrics."

McWilliams did not pretend to have the definitive answer to making the player happier. It is, she maintained, very much an open question, and one worthy of detailed discussion.

However, she was more decisive on one matter. Pulling up a slide of a chart depicting the average player lifespan on mobile platforms - descending rapidly with the passage of time - she said, "I maintain that this is crazy bullshit." She pointed to EverQuest as an example of a game that managed to build a lasting relationship with its players, and without the high churn rate that is basically expected by so many developers.

Speaking in the subsequent Q&A session, Teut Weidemann, senior online game superviser at Ubisoft, agreed wholeheartedly with McWilliams' stance.

"People forget that we're in the online games business," he said. "People see the metrics and only think about the monetisation. At Ubisoft we call players 'fans'. It makes you think differently."

14 Comments

shhh, the bean counters will not approve of this truism :)

Posted:10 months ago

#1

Andrew Ihegbu Studying Bsc Commercial Music, University of Westminster

473 183 0.4
Management in general would love to say the opposite in most realms as it makes their job much easier (read metrics, act on metrics, rinse, repeat) and diverts responsibility from the decision-makers (who are now just following the data) to their subordinates (who must have messed up implementation).

It's funny/timely that this should come out right after the Flappy Bird shutdown. The entire premise of the game was only a little more advanced than pong, but for odd reasons became incredibly popular. According to Nguyen's comments that is a good example of a game with great retention by not much enjoyment. (Addiction isn't fun)

Edited 1 times. Last edit by Andrew Ihegbu on 12th February 2014 5:29pm

Posted:10 months ago

#2

Brian Lewis Operations Manager, Aeria Games Europe

141 93 0.7
As we all know, '90% of all statistics are made up'. In the same vein, the meaning of most metrics are made up as well. Metric driven anything is only as good as the metrics (and the interpretation of them). This is true in any business, and often is the result of the use of stats/metrics to prove a point, rather than using them to determine the point.

If you have a preconceived notion, and only look for proof of that notion, then it is easy to find the supporting data (and ignore anything that doesn’t fit the notion). If, instead, you look at a wide range of data, and use it to determine a result (that includes all the data) it is more likely to be accurate. The same applies to metrics. If (for example) the assumption is that DAU is the key indicator of a healthy game, it is easy to generate numbers to prove that… up until the game dies. The same applies to all design choices, and to most metrics used by the industry. They’re only as good as they are valid… which is why they should always be questioned.

Posted:10 months ago

#3

Murray Lorden Game Designer & Developer, MUZBOZ

203 72 0.4
Great article, it's lovely to see someone with so much experience in the area raising some counter-arguments against metric driven design.

Of course - as long as metric driven design yields good profits - it'll remain popular. And it obviously has some useful purposes in practice.

But I'd hope that general game design intuition - with the goal of producing an engaging, fun, enriching, rewarding game experience, can remain near the forefront too!

Posted:10 months ago

#4

Nicholas Lovell Founder, Gamesbrief

210 254 1.2
I'm often thought of as a bean counter. I approve wholeheartedly.

Posted:10 months ago

#5

Nicholas Lovell Founder, Gamesbrief

210 254 1.2
Metrics are a tool. They are an incredibly useful tool that can help you identify where you have issues in a game, and to help you understand whether the thing you did to improve the game helped or made it worse. They are as useful as a designer's instinct and more useful than listening to the vocal minority in your forums or Twitter to help you make better games.

As always, though, the metrics are a target. A target is a proxy for an objective. If you shoot for the target, instead of the objective, you will get in all kinds of trouble.

Posted:10 months ago

#6

Paul Johnson Managing Director / Lead code monkey, Rubicon Development

959 1,758 1.8
I got blasted by the experts for a similar sentiment, but I still agree with it.

Metrics *can* give some useful numbers, but it's measuring secondary details just because the primary details aren't measurable. I think it's much better to engage with consumers as much as possible and try to get a feel for what should be improved that way.

Because once you have enough evidence of the problem, it really will be the actual problem and you can fix it with certainty. I have direct experience of this.

Edited 2 times. Last edit by Paul Johnson on 13th February 2014 9:11am

Posted:10 months ago

#7

Paul Johnson Managing Director / Lead code monkey, Rubicon Development

959 1,758 1.8
There's also an element of doing it "by the numbers". Games are meant to be an expression of what the developer feels is a good game, which includes making their own decisions.

Using metrics can indicate suggested changes, but when you stop thinking for yourself you lose a bit of soul. If you *only* want to make money, there are safer investment industries you can to move to.

Posted:10 months ago

#8

Claas Grimm Business Development Executive, Red Hot CG

7 11 1.6
"Love can't buy you money" - Motorhead, 1996

Posted:10 months ago

#9

Alfonso Sexto Lead Tester, Ubisoft Germany

846 732 0.9
Seeing how online rage runs havoc in the most elitist sector of "World of Warcraft" and "DOTA2" I wonder if that people actually is having fun,

I think "addiction" is the word in those cases.

Posted:10 months ago

#10

Christophe Danguien games developer

70 83 1.2
Did some people really need an article to know that....really ?

Posted:10 months ago

#11

David Serrano Freelancer

300 273 0.9
It's a mix of logic and emotion that goes into our decisions as game designers, she said. That's why we can never design by metrics.
On one hand, she's right. Metrics will never be a substitute or replacement for creative talent. But on the other hand, this is exactly why Chris Bateman wrote in 21st Century Game Design: "What is clear is that designers who design games solely to please their own sensibilities are unlikely to create genuinely mass market games."

Posted:10 months ago

#12

Gareth Eckley Commercial Analyst

88 68 0.8
I don't really like this bashing of games metrics. It's all very trendy, but the problems with metric assisted design are in the implementation. The underlying usage of maths to help understand the behaviour of systems is pretty well founded by now.

Metrics can only measure the things the developers want them to measure. An excellent example of how they can be subverted is with the recent Dungeon Keeper approval rating mechanism.

If you want to measure "love of a game" then you need to first analyse and then define the behaviours that are typical to highly invested players. This can only be done by a mixture of analytical and creative thinking and is where marketing and analytics can add real value to a product by assessing real player behaviour and tying that to in game event paths and numbers. This behaviour will always be game specific and require total honesty on the part of all the stakeholders.

In the above scenario, metrics can give extremely useful insights into game behaviour - or which player/design choices ultimately end up in player dissatisfaction. However, much like a parliamentary democracy, there's a lot of scope for the "facts" to be either biased to begin with, measure unimportant values or simply be ignored or re-interpreted as desired.

Assume that 10% of players have player housing. It is not the fault of the analytics team when someone in the creative side of the team decide that this means that 90% of players enjoy the fresh air more.

As Brian Lewis mentions above,

"If you have a preconceived notion, and only look for proof of that notion, then it is easy to find the supporting data..."

This is a highly prevalent attitude amongst people using metrics. They merely use it as another method to further their agenda.

Posted:10 months ago

#13

Laralyn McWilliams Studio Design Director, The Workshop Entertainment

1 0 0.0
There was no bashing of metrics in my talk, and I don't interpret the article that way either. I was one of the first designers to talk at conferences about integrating metrics into the design process, in fact, and I've been a proponent of them for years!

The focus of the talk is on the importance of balance. I've personally experienced decisions about game design being made based on metrics alone, and my point is that metrics only reflect behavior, which is half the picture. Emotion is the other half, and metrics can only vaguely hint at player emotion so we need a more holistic way to solve problems that incorporates both aspects.

On the other hand, you say: "If you want to measure "love of a game" then you need to first analyse and then define the behaviours that are typical to highly invested players."

Yes, but a key aspect of my talk is that invested behavior is NOT the same as happiness, and it is not automatically a measurement of "love of a game." It can be, for some players, but it's a dangerous assumption to paint every player with that brush stroke. Consumers continue with an investment past the point where they were genuinely happy--hoping it gets better, because it's a habit, because a friend is pressuring them, or other reasons. If your process assumes that just because he's still playing he's happy, you'll completely miss the fact that he's already begun the mental process of churn and by the time you realize it he may have turned a corner past redemption by any offers or win-back campaign.

Posted:9 months ago

#14

Login or register to post

Take part in the GamesIndustry community

Register now