Key questions to ask before hiring a QA vendor

Developers and publishers spend a great deal of time planning all aspects of their game's development. Often these companies reach out to third-party industry vendors to help lighten the load and stress.

One of the more common pillars that is partially outsourced is Quality Assurance (QA). That's because whether developers build QA into their plans from the beginning or not, unforeseen fires can start in production that need putting out.

Sometimes, game creators just need another set of eyes to help ensure their vision for the title is being realised. This is the one thing everyone in our industry has in common: the passion for creating great games that can be experienced by players all over the world.

If you're new to QA or come from a start-up indie studio, you might not be familiar with the process of engaging a vendor -- or even know where to start.

The purpose of this article is to provide studios who are navigating these waters with answers to common QA questions, highlighting the range of tools available to developers and those best suited to their production needs.

Mitigating the need for crunch

In life, even the best-laid plans sometimes go awry. Video game development is no different. Imagine yourself in this scenario: your game is moving out of alpha phase and you realise you haven't put nearly enough hours into your test plans as needed. You're running out of time before declaring beta or before meeting your certification deadlines.

It is always best practice to engage with third parties at the initial planning stages of your project

In this situation, you typically have three options:

1. Miss your deadline (and swallow the consequences)
2. Have your team pull all-nighters and work weekends
3. Engage a third-party vendor to ease the burden

The key is not to panic. Help is at hand. By outsourcing the testing process, you can instead focus on the creativity that makes your games great.

The larger QA vendors can usually ramp up their services in line with your project timeframe, and should be able to supply a team at short notice, allowing you to work on track and on budget to meet deadlines.

However, it is always best practice to engage with third parties at the initial planning stages of your project. This should allow both sides adequate time to plan and avoid pain further down the line.

Request for proposal

There are many elements to QA, including functionality QA, certification, hardware testing, PC and mobile compatibility as well as stress and load testing.

Before you choose a vendor or seek recommendations/testimonials from industry peers, you'll want to research the specific services available for your project.

If you've planned out your QA strategy and timelines, and are looking to see what services are available from external providers, one of the best options is to put out a request for information (RFI) and/or a request for proposal (RFP) for your business.

These are introductory documents that allow for publishers to learn more about how any vendor works. While these are common in engineering and art houses, they aren't seen as often as you would expect for QA. Essentially, RFP documents allow producers to solicit bids from agencies. Some common material included in these documents is:

• Information about the company itself
• Rates
• Examples of previous work or case studies
• Studio information
• "What would you do in this situation?" kind of questions

Firstly, be patient if you request examples of previous work, as quite often these agencies aren't allowed to share the names of titles or companies they have worked for due to non-disclosure agreements (NDA).

Often, there are sensitivities -- for various reasons -- whereby developers and publishers may not want the fact that a vendor worked with them to be in the public domain. However, they should be able to provide general information on instances where the vendor executed on a project with similar requirements. This could be in the form of a case study or project 'post-mortem' report where the client is unnamed or their brand is redacted.

Another request that producers can ask from potential agencies is their Business Continuity Plan (BCP). This is quite topical due to COVID-19: when the pandemic hit many agencies had to migrate most of, if not all of, its production to remote working. This also occurred for the majority of third-party games agencies.

Protection of your IP data is critical and working with a company with a proven track record in security gives you peace of mind for your game

These BCPs allow you to see what internal contingency plans these agencies have in the event of unforeseen issues, as a studio may temporarily cease production for any reason (long-term loss of power, or network capability, etc).

Most solutions revolve around transitioning work from one studio to another and providing SLAs on how soon it would take to get spun up. Also, you want to be satisfied that your game is in safe hands, so question the security provided by any vendor.

Protection of your IP data is critical and working with a company with a proven track record in security gives you peace of mind for your game. You should also consider the physical location of the vendor. If you are based in Europe, it might not be ideal to have your external QA work done on the west coast of the US. Working in the same time zone increases efficiency.

Alternatively, seeking a vendor that can provide a 24/7, 'follow-the-sun' model for continuous around-the-clock production may be needed.

Knowing what levels of service to expect

But what about the 'quality' in 'quality assurance'? How do you know you'll receive a satisfactory level of service to help improve the overall quality of your game?

Some common clarifications that any potential partner should be able to provide include:

• Internal defect KPIs: invalid bug rate (shows that staff are writing a majority of "actionable" defects) or escaped defect rate (if they are working on live titles and are responsible for them)
• An example of bug writing quality
• An example of test case creation or sample QA strategy
• Staff attrition
• What will my team look like?

For examples of test case creation or sample QA strategy, some companies may already have an effective strategy that they want a vendor to execute on, while some companies have just been focusing on engineering and internal planning and haven't gotten around to understanding "how to QA their game." It happens more often than you think.

Agencies should be able to assign higher level QA leads or a project manager to work with you on building out a robust QA strategy and test plan for your development lifecycle. In terms of what the team assigned to you will look like, this question comes up a lot and the answer varies.

Give as much detail as possible on your project

If you are looking for a team of five analysts who are experts in RTS titles, be sure to call this out early on in your discussions with your vendor of choice.

It's important to set expectations. Nothing damages a relationship more than saying "I wanted this!"... Then finding out that you didn't get what you wanted for any myriad of reasons. Agencies generally have a lot of experts across different game platforms and genres but, sometimes, these personnel are dedicated to other teams.

Any reputable vendor will provide transparency and communicate to you why they may want to "move Bob over to another project." As vendors, it's our job to provide the best expectations on both sides of the relationship.

Pick up the phone -- it's good to talk

If you've selected a potential vendor or two, I would suggest that you set up an NDA with them (if you haven't already!) , and schedule a call to discuss your needs.

These chats are good opportunities to explain your needs in real time, and allow for quick questions regarding scope and needs to take place in a relatively short amount of time.

From these calls, your potential partner should be able to provide you with a plan of action or proposal. These proposals usually include the following:

• Brief examples of previous work
• Alignment on what the ask is for your company
• Discussion of initial strategy
• Rates and timeline
• Additional business collateral on the service that you are requesting information on

It's also a good idea to ask for their experience and ability in creating bespoke, custom testing frameworks and also integrations that are specific to your existing systems.

In conclusion, entrusting an external team with your QA needs may seem like a daunting idea initially but there are many QA companies who specialise in quality video game testing services.

A little research and some careful consideration on your side will go a long way to the success of your project and, ultimately, the satisfaction of your players.