Devs shouldn't dismiss complainers despite what metrics say

"Metrics aren't everything" may be something of a theme at this year's Montreal International Game Summit. Following up on Jen Whitson's presentation earlier Monday, A Thinking Ape designer and producer Tayber Voyer added his own take on that sentiment, stressing that data, while helpful, only gives developers a part of the picture.

"Data-driven design is too shallow because you lose the context of the experiences you create," the Kingdoms at War and Pot Farm developer said. "With player feedback, you can learn and understand context. And context is what creates emergent behavior. And behavior makes your product great."

Much of Voyer's talk focused on the importance of incorporating community feedback into the decision making process hand-in-hand with data. Just as it's silly for developers to ignore the tools advanced analytics provide, it's similarly short-sighted to follow the numbers blindly without understanding why they're saying what they're saying. The advantage of bringing user feedback into the mix is that it can provide developers with explanations for player behavior they never expected.

"In all of the games I've worked on that have been successful--which is two of them--they all had a huge component of emergent design and emergent behavior," Voyer said. "We launched systems into the game that players took and used in ways we never imagined. And if we just looked at the metric, we would have never understood that."

One process Voyer endorsed for games-as-a-service developers was what he called a "minimum viable feature" approach. The idea is to hack a feature into a game with as little development resources as possible, just fleshed out enough to test a basic hypothesis. If it doesn't work, go back to the drawing board. If it works, devote some resources and make the feature a proper part of the game. He gave one example from Kingdoms of War that revolved around a new approach to the game's promotions.

"If you're the producer or a game designer in the project and you're not actively involved in the forums and actively interacting with the community, you aren't doing a good job"

Tayber Voyer

Voyer and the Kingdoms at War team hypothesized that limited time promotions would ramp up engagement and get players to spend more. But rather than build a daily promotion scheduling back-end to automatically cycle in and out these sorts of promotions, Voyer said they manually changed the promo conditions on a different "epic battle" on a daily basis for a full month. The initial results were mixed, but when they doubled the gold drops on the epic battle that already had the highest payout, they saw revenues jump. The data told them people wanted the gold drops, but when they made a new epic battle that paid out at the same level as the promo as a matter of course, they found it didn't perform nearly as well.

The explanation was found by going into the game's forums and talking to the players about why they liked the first epic battle. It turns out it wasn't the gold drops so much as it was the art, or the unique way players had to use an item to interact with the boss. To take advantage of that learning while still retaining the special feel of the limited time promotion, the developers gave the players the ability to start that epic battle with a guaranteed tripling of the gold payout, but only if one of the players had a special key that was a rare drop in the game (or available as an in-app purchase). This in turn produced some more unanticipated player behavior, as Voyer said they found groups organizing around the players who had these keys, with the keyholders receiving preferential treatment from their fellow adventurers once in battle. Since then, the team has been considering new social dynamics they could design the game around.

Of course, not all of the feedback games-as-a-service developers receive is positive, or even helpful. But even if people rail against a change as it boosts revenues, Voyer stressed that's no reason to dismiss their complaints, or assume they're lying about what they want. It just means you lack context to understand, Voyer said. And when the players aren't articulating exactly what the problem is, developers need someone on the inside who understands what they're actually saying.

"You need to have people to champion the players' voice," Voyer said. "If you're the producer or a game designer in the project and you're not actively involved in the forums and actively interacting with the community, you aren't doing a good job."

Above all else, developers should stop dismissing complainers just because the data (and sales) may appear to contradict them on the surface.

"They're important," Voyer said. "They're the people that pay for your game to exist. And even if they aren't a high spender, they're critically important to the ecosystem of your game. Every player is."