How mobile game devs are evolving test and launch strategies

Launching a successful new mobile game is increasingly risky and expensive. Yet, despite significant industry changes in recent years, most developers continue to test and launch new games in largely the same way as they did three to four years ago.

At Google Play I've had the chance to work with mobile developers on hundreds of game launches. I've seen first-hand how developers are evolving their development and go-to-market strategies in an effort to "de-risk" new launches (and where they're not, but probably should be). I hope that by sharing some of these insights you'll find ways to adapt and evolve your go-to-market playbook to make your next launch more successful.

Traditional launch playbook

First, what do I mean when I say that most developers continue testing and launching new games as they did in years past? Many developers today follow what I call the "traditional launch playbook".

"It's important to continuously evaluate processes as a means to an end and iterate where necessary, especially in a rapidly changing environment"

A more traditional launch playbook usually involves extensive internal testing in early development phases, followed by various stages of external testing prior to full launch (e.g. technical, retention and monetization testing). Most often, external testing consists of releasing a production version of a new game (vs a beta version) in specific geographies at various stages.

For instance, many developers with a more traditional playbook do technical testing in the Philippines, retention testing in the Nordics, and monetization testing in Australia or Canada. While the countries may differ here or there, these new games are usually tested in no more than five to seven countries in total, with deployment limited to higher-end device types due to stability and rating concerns. In general, retention and monetization testing lasts about two to three months, with game teams focused on tweaking the game to reach certain KPIs they believe offer insights into how their game might perform on a global scale at launch.

Does this sound familiar? If so, it's not necessarily a bad thing, as this process still works great for many. However, at Google Play, we believe that it's important to continuously evaluate processes as a means to an end and iterate where necessary, especially in a rapidly changing environment.

A new era

A few years ago, the mobile games industry was in its nascency and really taking off in terms of profitability. Between 2010-2014, low barriers to entry and the size of the potential install base drove thousands of developers to enter the mobile gaming space. During these "gold rush" years, games could be made and launched quickly with relatively fast development cycles, low development costs, and reasonable marketing budgets. You could launch a new game, see if it made any money in the market and then decide if it was worth investing in further.

Fast forward to the present day and the market looks very different, as does launching a top new game. Let's quickly review some of what's changed in recent years:

1. More competition. Launching a hit in today's more mature and crowded ecosystem can be very challenging. With more than 1M games offered on Google Play alone, consumers have more choices than ever. At the same time switching costs for players are high. As players become more deeply engaged and invested in the games they play--building up progression, characters, decks, alliances, resources, etc.--they are less likely to switch games and abandon their hard work (not to mention time and money invested). As a result it is getting harder and harder for new games to break through, and we're seeing lower turnover in the top grossing charts for games (at least in most Western markets, including the US).

2. Higher development costs. The resources required to properly test and launch a new game in today's market are significant. Not only are development cycles longer, but consumer expectations and demands of mobile products are also higher than ever, and they churn extremely fast if not satisfied (i.e. no more minimally viable product launches). Add this to the cost of longer title cycles and the need for increasingly sophisticated tech--to keep file sizes small, matchmaking quick, and keep up with trends like synchronous PvP and real-time chat translation--and the resulting costs to create a new game can be considerable.

3. Higher marketing costs. Today the marketing budgets for top new mobile game launches look more and more like those of console game launch budgets. Not only is organic discovery more difficult in a highly saturated market, but user acquisition continues to be extremely expensive, and the cost of the high quality tools, tech, and analytics needed earlier in a game's development cycle add up quickly for developers.

As a result of many of these shifts, we are seeing fewer new mobile game launches than in years past and more risk aversion from developers across the board (opting for safer bets versus moonshot bets).

Process as a "means to an end"

With so many changes occurring in the mobile games market, it's interesting that many developers continue to test and launch new games as they have in years past, but it's not surprising.

Process is an area where it's easy to stay static when other parts of the business change. That said, in a rapidly changing market, over reliance on process can be dangerous, as it can create a false sense of security (e.g., you might not even realize you're behind or that you have developed blind spots). In general, I've seen that teams who focus on process as a means to an end--where they are more focused on what they are trying to achieve versus how they exactly achieve it--are generally more adaptive, and ultimately, successful.

Evolving your playbook

While many developers continue to rely on the more "traditional playbook," more and more are evolving their mindsets toward releasing new games and innovating on the way they test and launch. I recently led a panel on "de-risking new game launches" at one of our Google events, where, together with developers from Electronic Arts, Wooga, Miniclip, Playrix, King, and Big Fish Games, we dove into this topic of process evolution and shared some best practices.

Here are five ways top developers are evolving their testing and launch processes in order to de-risk new game launches in today's market.

1. Continuously evaluating new games (and not being reluctant to pull the plug)

To mitigate many of the inherent risks around launching a new title, many game developers and publishers have shifted their focus toward investing in "fewer, better" games. To achieve this, they've evolved their internal processes around greenlighting new games and implemented more rigorous evaluation at multiple stages of development.

"Many developers are taking a 'guilty until proven innocent' approach to greenlighting new games"

A few years ago, it was common for there to be one early, internal greenlight for a new game before a game team had permission to take it to market (launching after that point was a given). Today, many developers are taking a "guilty until proven innocent" approach to greenlighting new games, requiring that they undergo critical evaluations at multiple stages and creating more opportunities for failure and feedback (launching is no longer a given at any point). For color, I've worked with game teams that have had to secure as many as five internal greenlights before bringing a new game to market. I've also worked with developers who have killed games that have been in development for a year--or in the late stages of retention and monetization testing--because their titles were performing below expected metrics.

While this process evolution seems relatively straightforward, implementing it successfully is difficult. It means creating an internal culture and climate, from the top executives down to the game teams, where it is OK to fail (faster the better) and where pulling the plug on a game that is not "up to snuff" is the norm, at any stage of the development cycle. Although this is hard to do in practice, against the backdrop of the current market, many developers believe it is better than absorbing the cost of launching (and maintaining) a mediocre game for years to come.

2. Getting external feedback earlier in development using open (or closed) beta

Due to the rising costs associated with launching new games, identifying bad ideas as early as possible has become critical for developers (again, if you're going to fail, fail fast). As such, many top developers are evolving their typical testing playbooks to get new titles to market for external feedback much earlier in the development cycle than in years past, sometimes even before full prototyping. Not only do they believe this is critical to identifying bad ideas early (allowing them to collect feedback, measure KPIs and make a call on whether to invest further or pull the plug), but many developers also believe that this early market feedback is critical to building titles with longer end games.

"Unlike with a production soft launch, testing via open beta means no public ratings or reviews"

To test new games externally earlier, many developers are evolving their technical approach to testing-- moving away from only testing production versions of their games to testing beta versions first, especially for tech and retention testing. One of the main reasons for this shift is that beta testing (open or closed) allows for more distribution control than a geo-locked production soft launch.

For instance, with open beta you can actually put a cap on a beta's size, allowing developers to collect early market feedback on a new game while also limiting public access to it at such an early stage. Plus, unlike with a production soft launch, testing via open beta means no public ratings or reviews--user feedback is private--which is a big advantage while you are ironing out issues in early stages of your game or doing bold A/B tests to see what really moves the needle for your KPIs.

While I've seen a huge uptick in the number of developers utilizing open beta for external testing, I've also seen developers dealing with big IP or well-established consumer brands relying more on closed beta for early external testing (due to license issues, etc). Today, these developers often set up external sites where interested users can apply for access to their closed beta test, before moving on to more traditional geo-locked soft-launch testing closer to launch.

After seeing the advantages open beta testing offered their game Cooking Craze,Big Fish Games now requires all new titles to run Android open betas before launch. Not only does this allow them to gather direct feedback from players, but it also allows them to test their new games earlier, bolder, and broader than they would have in a production soft launch. This means taking bigger risks with art style, gameplay mechanics, promotions, etc., and testing new games in more geographies and on more device types. The resulting benefits range from discovering strong performance in various markets, to exposing more unknown technical issues (and more accurately estimating impact of known issues), to fewer 1-star reviews at launch. For Big Fish Games, testing via open beta means minimizing the risk and mystery of how new titles will perform at launch, allowing them to better prioritize QA resources, optimize global UA channels and budgets ahead of launch, and release more confidently all around (with fewer surprises!)

Watch on YouTube

No matter your budget or size, testing new games via open beta is definitely a process evolution to consider. Plus, Google Play's new country targeting for beta releases feature allows developers to distribute their beta and production APKs to different markets at the same time, so you can test a beta version of your game in most geographies and move to a production soft launch in select markets before launch to confirm KPIs and build up a solid ratings pad.

3. Focusing on long-term metrics as predictors of success (and testing longer)

In recent years we've seen a shift in game development from a focus on driving downloads and monetization, to a world where engagement is the true KPI of sustainable business success (especially long-term retention and LTV). As a result, many developers are evolving their playbook to build in time and budget for much longer testing periods than in years past.

While there certainly isn't a "one size fits all" model when it comes to testing duration--depends on what you're testing, how much you need to optimize, etc--at a minimum, developers should be testing beyond day 30 metrics. Although D30 metrics are key indicators of longer term retention, to understand true user value you need to understand long term engagement and retention patterns (LTV is the long game). This means that even if your game is perfect, you should be testing for at least two to three months. On average most developers I work with test new games for six months, but often as long as 10 or 12 months for bigger titles (if not more). For games with big IP, it's especially important for developers to understand what happens to their users after D30 as IP can often mask bad metrics in the short term. People may stick around in the early game because they like the characters, but that doesn't necessarily mean you have good retention.

4. Shifting mindsets towards games as long-term investments

Five years ago, few developers thought a mobile game could be successful for five years. Instead they'd launch a game and start thinking about the next big thing. Today, games like Candy Crush and 8 Ball Pool have proven mobile games can have end games of five or 10 years. As a result, many developers are increasingly looking at games as a long-term investment and are shifting their focus from only investing in new titles to doubling down on existing hits. This means not only maintaining these titles day-to-day, but also significantly reinvesting in these titles to ensure that they are delivering new content, investing in Live Operations, staying fresh from a UI perspective, and keeping up with new trends. This not only helps engage and retain their existing players, but helps attract new ones as well. For many developers today, big updates are "the new launch."

Considering how competitive today's market is--and how expensive it is to acquire users--it's no surprise that many developers are focusing on supporting existing titles with established users bases over taking on the risks of launching a new game altogether. For similar reasons, we've seen large developers and publishers move away from annualized releases for franchise titles, in exchange for rolling out evergreen titles with big seasonal updates. Electronic Arts, for instance, has done this with their sports titles such as Madden, NBA Live, and FIFA. This is a big shift considering that FIFA had multiple annualized mobile releases before they launched FIFA Mobile last year. Now, for EA and many other developers, the decision to launch a new game in an existing franchise is fundamentally driven by the need to improve the underlying game tech/engine (versus a desire to launch something new).

If you have an existing game with a strong user base, it is worth thinking about how you can invest in making that title more successful through live operations and frequent content updates instead of only thinking about your next big hit.

5. Optimizing for retention before launch

Launching isn't just about the launch any more, it's about retaining users from day one. As a result, many developers are evolving their development playbooks to focus on maximizing retention in their game prior to launch. Many believe that to be truly competitive as a top new game today--against other hit games in the market--means ensuring long-tail content pipelines, social features and event plans are lined up and resourced from the start (versus "MVP" launches that only have 25% of the features you'd hoped to have at launch). That said, upfront investment into these features prior to launch can have significant implications on budgets, headcount, and resources.

Here are the top areas where I see many developers evolving their processes when it comes to optimizing their games for optimal retention:

• Focusing on quality and performance prior to launch. Strong app health and performance metrics (e.g., crash rates) are key to preventing early player churn and minimizing bad reviews at launch (50% of 1-star reviews mention stability and bugs). As a result, many developers are testing new game performance on a much wider range of devices prior to launch. This is particularly important for developers on Google Play as the Play Store algorithm is being tuned to promote quality and engagement, with key indicators of this being a game's Android Vitals performance, retention, and ratings.
• Making sure they have enough content at launch to sustain engagement and prevent churn. Rather than making levels harder to slow players down and risk churn, many developers actively test how quickly their players consume content and move through levels during their beta, and make enough content to sustain them. Remember, you can also recycle content with events!
• Building social features into their games prior to launch. Social features are critical to engaging and retaining users in any game, including in a new game at launch. The more engaged users are, the less likely they are to churn. Alliances and alliance wars should be built in for day one, as should live services and events. This means investing in a team dedicated to LiveOps before launch.
• Not being afraid to "waste" good events during beta. Developers need to know what KPIs their LiveOps move, how they impact their player behavior, and how their teams can stand up to a demanding LiveOps cadence. Often this means testing their LiveOps (and their team's ability to deliver) before launch as if it's the real deal. For instance, June's Journey by Wooga is a game that was extremely demanding from a content production perspective. A month before launch--after a seven-month soft launch where they ironed out stability and KPIs--Wooga orchestrated a full "dress rehearsal" of their launch where they treated their game as if it was live (weekly releases, marketing reviews, social media, etc.). By not holding anything back in their "rehearsal," they created critical opportunities for their teams to practice and understand the demands of their full LiveOps plan, and ultimately better prepared themselves for the real launch of their new game.

If you haven't re-evaluated your go-to-market playbook in the past few years, it's a great time to dust it off and look at the new trends and tools available. I encourage you to try some of the examples I have shared and see if you can identify areas to improve your process or avoid blind spots.