We need to talk about tech tests (and how to improve them)

Imagine this scenario: you found your dream game company, they have the perfect opening for you, the job spec is a great fit to your skills and experience, the project is cool and exciting. You decide to prepare an application, you brush up your resume and write a tailored cover letter explaining why you’re the best candidate for this role, and send it their way.

A few days later you get on an intro call, you go through your profile and learn more about the company. Everything seems to be going great. Finally, as the hiring manager explains the hiring process, you learn that the next step, before talking to the engineering team, is to complete a timed technical test.

You probably don’t need to imagine this, as every engineer working in the game industry, at some point in their career, had to endure a lengthy technical test as part of a recruitment process.

Tests are in my opinion an ineffective way to conduct tech recruitment. In this article I’d like to explain why they provide an unclear and biased picture of the candidate, the burden they put both on the applicant and your hiring team, and a better way to assess your candidates that is more fair and inclusive.

A missed opportunity

Technical tests come in multiple variations. Some companies will give you a problem, asking you to solve it using their preferred language and tools, and following a list of requirements before sending the source files back. Others will ask you to develop a small game from start to finish, based on a design spec they provide. Some studios will provide a starting project with a few bugs baked in, and request you to further develop it and bugfix.

In all of those cases, the end result is some pieces of code, with little context and without an understanding of the thought process of the candidate. This feels like a less than optimal way to assess their technical skills, easily leading to misunderstandings and confusion.

Writing code is not even the main thing you should expect from an engineer. Code is just the final result of a series of decisions and considerations, and those are the areas you really want to focus on. Why did they pick a certain pattern and paradigm? Have they considered alternative solutions, and if so, what were the advantages and disadvantages of each? How would they further develop this if they had more time? All of these questions are left unanswered, leaving the reviewer in the dark.

If the goal is to learn more about the candidate and to understand how they solve problems, why not just do this together in person or on a call? It is certainly a missed opportunity to get to know the candidate and the way they work.

Writing code is not even the main thing you should expect from an engineer. Code is just the final result of a series of decisions and considerations, and those are the areas you really want to focus on

From the candidate perspective, it is often hard to accurately assess what the interviewers might expect, and what their preferences are. Of course they can ask for clarifications, but they might refrain from doing so as email communication can be slow and the deadline is fast approaching. Often applicants will attempt to provide some notes and references by adding comments to their code, or including a text file.

Most importantly, companies that rely on technical tests are focusing on the wrong thing. The best engineers are the ones that can communicate effectively, discuss the requirements while asking the right questions, and challenge their interlocutors to achieve the best solutions. They are always ready to learn new technologies, to adapt to unfamiliar environments and challenge themselves. None of these traits will emerge from a technical test.

Tests are also a burden on your team

Technical tests are often seen as an easy and quick way to filter out candidates that do not possess the right technical skills. Rather than spending time interviewing each and every one of them, companies shift the burden of proving their worth to the candidates themselves, before proceeding with the recruitment process.

In my experience this is far from the truth. Companies tend to underestimate the amount of time and effort required to properly evaluate tech tests. This normally leads to two possible outcomes, both less than ideal.

In the best case, engineering teams will need to reduce their workload to support the hiring pipeline, delaying their delivery and the overall project. This is rarely planned, which leads to frustration and missed deadlines.

In the second outcome, which I would argue happens more often, engineers will quickly skim through the technical tests without dedicating the right attention to it. This ultimately leads to candidates being unfairly evaluated, despite their hard work and time spent, and the company missing out on potentially great hires.

They are not inclusive and unfairly put people at a disadvantage

Job seekers tend to apply to multiple companies at a time, as there is no guarantee that a job application will result in an actual interview. If each of those would require going through a test, that would quickly add up to weeks of unpaid work.

Many candidates will have a job, family, dependents, and other life commitments. Asking them to spend multiple evenings and weekends on an assignment can be a significant burden, and honestly not very inclusive.

You are essentially giving an unfair advantage to people with more free time and fewer obligations

One (sadly) common opinion is that if the candidate is not willing to put the time and effort to do the test, they were not really interested in the position anyway. This is both unfair and inaccurate.

Different people will be able to dedicate different amounts of time on the test, which has a direct impact on the quality of the submission. You are essentially giving an unfair advantage to people with more free time and fewer obligations.

This leads to biased results, ultimately preventing entire groups of people from entering the game industry, or forcing them out to other areas in tech as they get older.

A better way to test candidates

As I covered some of the issues that traditional technical tests have, I would like to propose a better way to assess the engineering skills of a candidate.

Rather than giving some homework to your candidate, a more effective way is to sit (in person or virtually) with them, setup a shared development environment, and explore a problem together.

The first step is to construct a scenario and present it to the candidate, outlining the current resources, constraints and the desired outcomes. Allow enough time in this phase for them to analyse the requirements and ask questions, as it will shine a light on how they approach problems and find solutions. What questions do they ask? Do they challenge the current constraints based on their past experience? For this purpose, the ideal scenario has to be open enough for them to find interesting and original approaches.

Then, it’s time for the candidate to get their hands dirty and develop the solution they proposed. This phase will look different depending on the actual job spec. It might require the candidate to develop a behaviour tree in Unreal Engine to control a group of NPCs, writing their AI behaviours and testing on a project you provided. They might navigate the AWS dashboard (or even better write a Terraform template) and prepare the infrastructure for the game backend they proposed, provisioning VMs and databases on a sandbox environment you provided. Or they might spend some time profiling a game, identifying bottlenecks and preparing a report of what steps are necessary to achieve a stable framerate.

In all of these cases, it is essential that you do enough preparations to provide the candidate everything they need to get right into it, without wasting time importing resources, creating accounts or installing software. For this reason, it is ideal to provide a computer (or a VM if done remotely) with everything already setup and ready to go. In addition, make sure to clarify in advance how much time will be allocated for this, so the candidate can plan their time accordingly.

Do not forget to ask the candidate in advance if they need any specific accessibility arrangement, such as screen readers or alternative input devices.

Keep in mind that every candidate is different, and many people might feel stressed or uneasy working while others stare at their screen, especially during an interview. Consider asking if they would prefer to have some time alone while they work on their solution, checking in from time to time. In most cases it should be allowed and expected to search for resources online, as it is an integral part of every software engineer’s work.

Some studios are already taking a similar approach, and from what I heard they’re satisfied with the outcome. I was also personally involved in a test like this as a candidate, and I found the experience fair and engaging.

Looking for a new job can be a very stressful time, bringing a feeling of uncertainty due to the eventual rejections and the time and efforts required. Often there might be a lot at stake for the candidates and their families. Because of this, it’s our responsibility as hiring managers to make the process as fair and inclusive as possible.

As the industry goes through multiple waves of layoffs, make sure to review your hiring practices and recruitment process to provide a better experience for everyone involved.