Devs shouldn't dismiss complainers despite what metrics say

Kingdoms at War producer says community feedback key to making data-driven design make sense

"Metrics aren't everything" may be something of a theme at this year's Montreal International Game Summit. Following up on Jen Whitson's presentation earlier Monday, A Thinking Ape designer and producer Tayber Voyer added his own take on that sentiment, stressing that data, while helpful, only gives developers a part of the picture.

"Data-driven design is too shallow because you lose the context of the experiences you create," the Kingdoms at War and Pot Farm developer said. "With player feedback, you can learn and understand context. And context is what creates emergent behavior. And behavior makes your product great."

Much of Voyer's talk focused on the importance of incorporating community feedback into the decision making process hand-in-hand with data. Just as it's silly for developers to ignore the tools advanced analytics provide, it's similarly short-sighted to follow the numbers blindly without understanding why they're saying what they're saying. The advantage of bringing user feedback into the mix is that it can provide developers with explanations for player behavior they never expected.

"In all of the games I've worked on that have been successful--which is two of them--they all had a huge component of emergent design and emergent behavior," Voyer said. "We launched systems into the game that players took and used in ways we never imagined. And if we just looked at the metric, we would have never understood that."

One process Voyer endorsed for games-as-a-service developers was what he called a "minimum viable feature" approach. The idea is to hack a feature into a game with as little development resources as possible, just fleshed out enough to test a basic hypothesis. If it doesn't work, go back to the drawing board. If it works, devote some resources and make the feature a proper part of the game. He gave one example from Kingdoms of War that revolved around a new approach to the game's promotions.

"If you're the producer or a game designer in the project and you're not actively involved in the forums and actively interacting with the community, you aren't doing a good job"

Tayber Voyer

Voyer and the Kingdoms at War team hypothesized that limited time promotions would ramp up engagement and get players to spend more. But rather than build a daily promotion scheduling back-end to automatically cycle in and out these sorts of promotions, Voyer said they manually changed the promo conditions on a different "epic battle" on a daily basis for a full month. The initial results were mixed, but when they doubled the gold drops on the epic battle that already had the highest payout, they saw revenues jump. The data told them people wanted the gold drops, but when they made a new epic battle that paid out at the same level as the promo as a matter of course, they found it didn't perform nearly as well.

The explanation was found by going into the game's forums and talking to the players about why they liked the first epic battle. It turns out it wasn't the gold drops so much as it was the art, or the unique way players had to use an item to interact with the boss. To take advantage of that learning while still retaining the special feel of the limited time promotion, the developers gave the players the ability to start that epic battle with a guaranteed tripling of the gold payout, but only if one of the players had a special key that was a rare drop in the game (or available as an in-app purchase). This in turn produced some more unanticipated player behavior, as Voyer said they found groups organizing around the players who had these keys, with the keyholders receiving preferential treatment from their fellow adventurers once in battle. Since then, the team has been considering new social dynamics they could design the game around.

Of course, not all of the feedback games-as-a-service developers receive is positive, or even helpful. But even if people rail against a change as it boosts revenues, Voyer stressed that's no reason to dismiss their complaints, or assume they're lying about what they want. It just means you lack context to understand, Voyer said. And when the players aren't articulating exactly what the problem is, developers need someone on the inside who understands what they're actually saying.

"You need to have people to champion the players' voice," Voyer said. "If you're the producer or a game designer in the project and you're not actively involved in the forums and actively interacting with the community, you aren't doing a good job."

Above all else, developers should stop dismissing complainers just because the data (and sales) may appear to contradict them on the surface.

"They're important," Voyer said. "They're the people that pay for your game to exist. And even if they aren't a high spender, they're critically important to the ecosystem of your game. Every player is."

Related stories

Outsourcer suing Gameloft over Asphalt assets and trade secrets

Vietnamese studio Glass Egg claims employees created assets without company's knowledge

By James Batchelor

Supercell sales, earnings down in 2017

Clash Royale studio posts lowest numbers in three years, still brings in $810 million profit on more than $2 billion in revenue

By Brendan Sinclair

Latest comments (3)

Adam Jordan Community Manager, Ubisoft4 years ago
Can't have the positive without the negative or to put it more correctly, without the negative, how can you define and explain the positive?

However it's a two way street, just like a lot of things. Games-as-a-service communities are a double edged sword. You can have a community with quite a few positive members, you can also have members that constructively voice their negative opinions but both sets of members will always be drowned out by the ragers and complainers and they are the ones that take focus from the real feedback.

In all communities I am involved in, I have always tried to teach the community that raging won't get them anywhere or heard more. However creating a topic with a well thought and executed explanation of why they feel a certain feature or item within the game "sucks" or needs improving will have a better chance. Granted that has helped but generally you will still get "X Company is LAZY, FIX UR DAMN GAME" or my personal favourite "Game won't work, fix it or I quit" (Which later turns out to be an issue with their PC and not the game)

HOWEVER, yes developers need to step up their game and listen to their communities. They are the key to success since after all, they are the ones playing and paying. Of course that's not to say, put everything they say into the game but like the example given in the article. If you do Daily Deals where players can get a discount on items for 24 hours...put in the effort and listen to what they want. Heck just a weekly deal which is done by a poll.

Regarding Producers interacting with the community or other developers for that matter. That one should be a personal choice and dependant on the hierarchy within the company. The problem with other developers except for the Community Team (and Manager) and QA; is that the community will come to expect other developers to be as active as the Community Team.

Personally it should be Community Team and QA as most active, with everyone else diving in when they have spare time or need to address something such as the direction of the game (Producers), game balancing (Designers) or In-game store (Store Manager)

Other than that, I have to agree, complainers are as important as the non-complainers, however those complainers do need to realise that in order for them to be noticed, their complaints should be constructive and not spewing hatred/insults (Granted not all of them do that but sadly there are some out there that do)
1Sign inorRegisterto rate and reply
Richard Vaught Studying B.A. in Game Design, University of Advancing Technology4 years ago
What a lot of people tend to forget is that refinement is by necessity a destructive process. It doesn't matter what you are refining, it is always destructive. The negative commentary is essential to the refinement process, it is what chips away at the crap in the object in question so that precious gems hiding beneath the surface can be found. If you are dismissing the complainers you are trying to polish a lump of coal without even realizing there is a diamond in your hand.
0Sign inorRegisterto rate and reply
Constructive Feedback is very valuable, unfortunately you usually have to wade through a lot of crap to find it.
1Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.