The Class Of Quality Assurance

Sony's Studio Liverpool alumni discuss whether QA really can be a launch pad into the upper echelons of the games industry

Quality Assurance, or game testing, has a bad reputation in the games industry. In some ways it's justified: the working conditions can be grim and the pay incredulously low. But, more damaging and dangerous, the bad reputation is often promoted and propagated by in-house attitudes towards QA that brands staff "minions"; low-level, disposable work-horses who'll do the most mundane of tasks for hours on end. As a microcosm for the way that society at large treats its lowest paid - and often hardest working - members, it's apt. And it's unfair.

"Doing the same things over and over again... mentally I think you have to be really strong to get through it and make sure that you're paying attention"

Rob Karp, Sony X-Dev

During my own time in Quality Assurance, across two years at Studio Liverpool from 2008-2010 I had the chance to see the truth and the lies of the QA myth, and found it a vital stepping stone for my subsequent role as a game journalist for Edge magazine and beyond.

First, the truth: It's low paid. It's not glamorous (you may have to spend an entire evening, or week, scrolling through menus or rebooting discs, for example, and make no mistake: your eyes and thumbs will take a battering).

The lies? QA staff aren't disposable minions; they're quite often the most passionate game players and consumers you'll come across, hence their ability to endure painfully repetitive shifts, and the right studios should acknowledge and capitalise on this. Another falsehood is that QA staff are largely uneducated, disengaged drones. Many QA staff are highly educated, and most (if not all) aspire to progress through the industry from QA and many, like David Deeble, now a designer at Sumo Digital, achieve that dream.

Deeble got his start as a tester following a stint in video game retail, and found it a mostly encouraging experience. "I was super-excited at landing my first job in the industry and had so much damn fun in the process," he gushes. "I still have that same excitement and enthusiasm today, I mean there aren't many jobs out there where you literally can't wait to get into the office every morning."

Deeble moved to the ill-fated Bizarre Creations to continue his testing career before making the leap to game design at Sumo, but it was his time at Studio Liverpool that seems to have left its biggest mark, building a solid foundation for his career. "I was at SCEE back in 2007, so this was when Studio Liverpool was still around," he says. "[There were] loads of departments, from First Party QA, Format QA, Third-Party QA, a localisation team, a video department... not to mention an entire dev studio. My department alone had at least 40-50 staff and that was just the night-shift."

Another evangelist for QA is Thomas Smith. Now working as a tech artist at Ubisoft's Massive studio in Sweden he, like Deeble, made a horizontal move from Studio Liverpool to Bizarre Creations, finding the more connected experience of working alongside a dev team ("embedded testing" as its widely known) an important experience. "Internal QA departments are critical to a studio," he says. "They create work as well as verify incoming work from external departments for the rest of the teams, they stabilise those important press and trade show builds as well as the end product. Localisation, user feedback, tools, distributing builds to external studios... the list goes on. If you didn't have great people in control of all that the project would come off the rails quickly."

"When I first joined QA, there were two other women on my shift and 50-plus men. At times I did find it difficult"

Michelle Tilley, Studio Liverpool

For people like Deeble and Smith, QA acted as a launch-pad into the nitty-gritty of game development, and it clearly primed them with a knowledge of the processes they'd later be working with and around.

Others, like Rob Karp - now a producer at SCE's XDev Studio Europe - have found room to manoeuvre within the very building where they enlisted as testers. Karp's current role has him jet-setting around the world, hand-picking the next big things for Sony to take under its wing as publisher, but his entry-point into this high-powered, high-level role was anything but glamorous: "One task I had [as a tester] on Formula One 2005 on the PS2," he reveals, "was to check the collision detection on the barriers for each track. I would boot up each track and drive along pushing into the barriers on the left side the whole time, to make sure the car didn't get stuck or fall through the world, I'd then repeat this down the right hand side of the track... It was as thrilling as it sounds."

While Deeble and many like him host a seemingly bottomless well of enthusiasm for simply being around games, Karp's view is a sharp contrast, with no illusions that it can be a soul-crushing role. "It's not easy to play the game for three, six or nine months," Karp adds. "Doing the same things over and over again... mentally I think you have to be really strong to get through it and make sure that you're paying attention, because the last thing you want is to miss a bug."

The mundane nature of some of the tasks you're handed in QA is something I know all too well. I've sang my way through Singstar, played infinite rounds of the same general knowledge quiz in Buzz! (I'm none the wiser for it) and completed inhuman amounts of foreign language playthroughs on a range of titles good and bad. One of the benefits of such grind-work is that it familiarises you with the evolution of builds and the general workflow of a development team as you watch a title mutate towards master.

Karp agrees: "It did give me a great grounding in part of the development process, it let me see and understand the transition from versions of the game that were sent for familiarity, to alpha, to beta to master. In turn this helped to give me a good overview on scheduling, particularly when it comes to the QA phase, but also before that, because I've seen the number of issues that implementing certain features late in the development cycle can cause; which areas are the highest risk."

Michelle Tilley, Planning Release Coordinator at Studio Liverpool, is another example of a tester graduating to greater things within the same four walls she started out in. "When I was a tester, I sometimes got frustrated by the lack of direction - I didn't know exactly where I could progress from QA," she explains. "But this became clear when I had the chance to shadow the Planning and Release team and when a job became available, I applied and was lucky enough to be given a place on the team."

Tilley's presence here, as the lone woman out of my interviewees, reveals another truth of QA: the disproportion in male and female employment figures. Tilley is open about the intimidation of such a skew in gender representation: "When I first joined QA, there were two other women on my shift and 50-plus men. At times I did find it difficult, I worried that people didn't take me as seriously as I thought they should."

"There was definitely a feeling of isolation and disconnect, not just from the company itself but with the developers you were supposed to be working with"

David Deeble, Sumo Digital

Karp identified the same issue but is adamant there's no discrimination at play; it's all about the applications. "Standards for getting into QA aren't the highest," he says, "but very few women applied and they were held to exactly the same standards as the men that applied."

"I think this pushed me to try harder," Tilley chips in. "To take on more responsibility and to constantly think of ways we could improve and change our processes for the better. It also gave me thicker skin and the motivation to strive for another job."

Encouragingly, Tilley and Karp both saw the tide changing even in the short time since they started out in QA. "It is still an industry dominated by men," says Tilley, "though, I do feel this is changing. I think this may be in part because of marketing heavily focusing on men or the simple fact that a game development career isn't something advertised to young men or women. I don't think it should be a simple thing of 'hiring more women' but getting more young women interested in a career in the games industry [in general]."

"I think the proportions are starting to get better, albeit slowly," adds Karp. "Both in QA and the broader industry. I think it's still a male dominated industry, across all sectors from gamers to  developers, and I think it is super important to hire the best person for the role irrelevant of their sex, whether their sex is disproportionately represented in the industry or not. "

These four examples already demonstrate QA to be more than a dead-end role for minions, work-horses, drones. But QA itself can (and with a little more encouragement from studios in terms of pay and inclusion should) be an attractive career in itself. Take Dan O'Shaughnessy, who came fresh from degree-level study of games into QA and worked his way through a range of exclusively QA-focused roles. "My first position at SCEE Liverpool was in the Format QA department as a summer contractor", he explains. "From there I moved to First Party QA as a Functionality Tester, and eventually became a TRC (Technical Requirements Checklist) Tester. I had the opportunity on a number of occasions to go on-site to work with development teams, like Evolution Studios in Runcorn and Guerrilla in Amsterdam, which allowed me to gain an insight to how things differ from studio to studio." O'Shaughnessy's experience speaks for itself, and clearly spoke to Ubisoft, who has hired him to lead testing procedures, on-site, for its Abu Dhabi team. O'Shaughnessy's progress is also a valuable lesson for videogame degree graduates disheartened by the prospect of an entry-level role after full-time study.

It's interesting, and perhaps points out a flaw in today's QA structure, that none of my interviewees managed to move into the creative development field at the very studio that was their home for so long; each either stuck to the more formal procedure-based roles of QA or jumped ship altogether. I certainly didn't find a creative outlet or opportunity during my stint in QA, but times and attitudes change and, hopefully, a heightening awareness of QA's importance and the quality of many of its staff, will catalyse a shift from within.

Deeble identifies a reason that may be to blame for the blockade often facing those with more creative ambitions - the size of the workforce: "It was easy to feel very small in a place like [Studio Liverpool]. We tested games from studios all over the world, so direct contact with developers was incredibly rare and even then was usually handled by a Lead Tester. Contact with the studio upstairs from us was practically non-existent, even when working on their games. I think there was definitely a feeling of isolation and disconnect, not just from the company itself but with the developers you were supposed to be working with."

"Nobody wants to be told there are bugs in their work. There is a tendency to forget just how crucial a good QA team is in shipping a solid product"

David Deeble, Sumo Digital

It wasn't until Deeble made the move to Bizarre and its more close-knit environment that he "learned a tonne" about actual design, interacting with staff from all major disciplines, from programmers to artists. The experience at smaller studios with more intimate headcounts likely equates to a more personalised experience and therefore a chance to make more of a mark on the development and QA process. It's an industry-wide phenomenon we've all heard before: when a studio grows too large many of its voices and faces get lost in the crowd and the sense of community, of belonging, can be lost irretrievably.

The stories of Deeble, Smith, Karp, Tilley and O'Shaughnessy are all unique and exciting in their own right but what they share in common, besides getting their start in the same QA workforce, is an insider's view that QA deserves more kudos. As Karp points out: "There's a big problem with the whole of the QA environment and that is that it's their job to find problems, which in turn causes a negative attitude. For me, the closer that QA works with the dev team the better; there's nothing worse than testing sections that get cut from the title or just aren't ready to be tested... it's such a waste of time."

Deeble backs up the sense that, because QA deals in problems, it's staff are often viewed as problematic: "Nobody wants to be told there are bugs in their work, right? Especially at 2am the night before submitting a build. I think there is a tendency to forget just how crucial a good QA team is in shipping a solid product. I can't speak for other companies, but here at Sumo we focus a lot on talent and ability. A good example is during development of Sonic & All-Stars Racing Transformed we hired staff to form an embedded QA and balancing team. Some of those guys showed real promise so we hired them full-time in Junior Design and Production roles."

QA clearly has its problems, both in terms of reputation and its working environment, particularly at larger studios, but if the voices in this feature - and the closing words below - point to one thing, it's that if you're aspiring to enter the games industry, it's not a bad place to start. And QA staff are often much more than "minions".

Famous last words:

David Deeble, designer, Sumo Digital: "Don't assume that you can waltz into a company and use QA as a stepping-stone to fulfil your dream role. Never lose sight that you're there to do a job, be professional, be the best you can and take the opportunity to talk to people and learn about the game development process. If you decide you want to move into other roles, hone your craft in your spare time and most of all be patient."

Thomas Smith, tech artist, Massive: "I would say if you do know what it is you want to do in the industry then go and study for it, don't join a QA department. If you don't know what you want to do but feel you like the idea of working with computer games I would say don't do it. You will have a bad experience getting stuck working on something you don't like and end up leaving in six months. If you have a plan and you need industry experience to make it happen then QA is the right place to cut your teeth but there is going to be a lot of hard work." 

Rob Karp, producer, XDev Studio Europe: "It can definitely be a thankless role, I try to make sure that the QA teams I work with now don't feel that way, but the problem is that QA is often just a small portion of the total time you're working on a game and when the QA testing is occurring it's more than likely that the team and producer are going to be very, very busy, with the QA often slipping down the priorities.... It's only a dead end job if you let it be one, it's so important to show enthusiasm and initiative, once you're ready to move on, it's up to you to make it happen. But be warned: don't just expect opportunities to come your way for simply doing your job. Over 50 per cent of the producers/senior producers from X-Dev Studio Europe started in QA. Be prepared to work long hours, don't expect for the team and producer to be happy when you find a must-fix issue the day before submission... "

Dan O'Shaughnessy, First Party Quality Assurance Tester, SCEE: "It should definitely be considered a gateway into the gaming industry. Think about where you want to go in the industry and never stop working towards the career you want to pursue... I have definitely benefited from starting my career in QA, but when I started at SCEE, I was hoping to work my way up the ranks and eventually become a SCEE programmer. Unfortunately it didn't pan out that way for me, but that's not to say that's always the case."

Michelle Tilley, Planning Release Coordinator, SCEE: "I would encourage anyone to join QA as a way of joining the industry. As long as you're passionate about games and have a willingness to learn, you can move into any job position you want. A lot of new starters seem to automatically want a job in production, but there are so many different roles at varying entry-levels the opportunities are endless. For my role it gives me a unique perspective, as I understand both the importance of scheduling release to hit a target date and creating a good user experience. Funnily enough, basic skills embedded into any software tester such as attention to detail, problem solving and working as a team can be applied to most jobs.

More stories

Zordix acquires Maximum Games for $42m

Purchase of publisher/distributor includes additional earnouts of up to $30m

By Jeffrey Rousseau

CD Projekt sales up but GOG struggles

Third quarter revenues up 38% as Cyberpunk and Witcher help offset digital storefront's ongoing losses

By Brendan Sinclair

Latest comments (18)

Arran Oakes Head of Production, GamingIQ, IQPC Exchange8 years ago
Very interesting article, some good myths busted here - although a different twist to consider is from the upper echelons of QA's perspective; people who passionately believe in the critical role QA plays, who want to encourage people to stay within QA, to rise up the ladder and develop their career within it, rather than just seeing it as a stepping stone.

Also interesting to observe an increase in the amount that QA is being reported on in 2013 - in previous years you'd rarely see an article so in-depth about QA, but now we have stories like this, Tulay's QA blog series over on Gamasutra, and of course the world's first ever event dedicated solely to QA & localisation in the gaming industry. Seems there is a rising trend here!
2Sign inorRegisterto rate and reply
Bruce Everiss Marketing Consultant 8 years ago
Several senior and successful people in the UK industry had Codemasters QA as their first job.
0Sign inorRegisterto rate and reply
Adam Campbell Product Manager, Azoomee8 years ago
As I've discovered myself - QA not only can be a great stepping stone, it can be a highly technical, progressive and complex role in itself - exposing you to a wide variety of technologies and responsibilities. Pay can also be great, though this really depends on the organisation and/or level.

The QA strategies and approaches are every bit as important as the rest of the development process and just as rewarding. I do find it interesting too how QA has recently been pushed into the spotlight but that's definitely a good thing. Products are all about the quality.

On the employment side of things, hopefully those that have been known to treat QA staff badly or use them as cheap, expendable resource will re-evaluate what QA means for their products and consider the human factors involved in employing people equally as passionate about the industry and needing a solid career in their lives. Well treated, well paid staff tend to put more time, effort and passion into what they do.

Edited 1 times. Last edit by Adam Campbell on 29th April 2013 11:21am

3Sign inorRegisterto rate and reply
Show all comments (18)
Paul Johnson Managing Director / Lead code monkey, Rubicon Development8 years ago
I think this is a dishonest dangle tbh. It is possible to join as the mail hop and rise to be chairman of the board, but I'm sure the reality is that you be a mail hop for a couple of years and then leave having reached the dizzying heights of head mail hop.

You could just maybe use the shoe-in to get your game designs in front of the right eyes, but if you want to be a programmer or artist then go get some skills in that instead. It's easy now than ever to teach yourself enough skills to get a junior position in the role you actually want.

QA is worthy job and I don't see why it's often used as a foot in the door. Maybe you should concentrate on getting to head of QA as a career path.
1Sign inorRegisterto rate and reply
Bruce Everiss Marketing Consultant 8 years ago
@Paul Johnson

For once I disagree with you.
The video game industry is a village, yet huge numbers of people want to get into it. It is the new rock and roll. Yet most of these aspirants are just game enthusiasts and have no idea what the industry is really like. A QA job enables them to see the true nature of the industry whilst giving them real game development skills and knowledge. It also enables an employer to get a good look at people to see whether they are worth investing in.
At Codemasters we recruited a very high calibre of person into QA. Many of them moved on to more senior roles in the company, I recruited several of them when I set up a social marketing department. Operation Flashpoint had a level editor. Codemasters QA used this to create a fantastic add on pack that we made a lot of money from selling!
As I said previously several ex Codies QAers now have senior jobs in the industry. And quite right too, they worked to get there on merit.
1Sign inorRegisterto rate and reply
Seb Downie Producer, Guerrilla Games8 years ago
I too came through Functionality/Localization testing in Scotland. Fun times :)
0Sign inorRegisterto rate and reply
Dan Lowe 3D Animator, Ubisoft Montreal8 years ago
I think you have to treat each individual on their own merits. I've worked with some of the guys in this article and it was clear that they were smart people who deserved a shot in roles outside of QA, but equally I think a lot of unskilled people get elevated outside of QA simply because they've been there for a few years and they're seen as having paid their dues.

If young developers aspire to roles in programming, art, design or production there really is no substitute for getting a solid education in that discipline. It doesn't need to be a degree, but banking a couple of years of solid work in your chosen discipline is far more valuable than getting a QA job and hoping your connections will lead to a promotion.
2Sign inorRegisterto rate and reply
Isaac Kirby Studying Computer Games Development, University of Central Lancashire8 years ago
I took a job in QA in my gap year. It tought me alot of the "harsh realities" of a first full time job. Coordinating workloads, how to write a damn good bug entry (just implemented new templates at my current placement year at Fujitsu) and much more.
As i moved on to my BSc in Games Development i realised that QA had given me a thirst to improve on myself, and made me very aware of "how it will look to a punter" in many circumstances. It also gave me my passion for AI, i was sick of being told "thats it, it wont get better".
I would thoroughly recommend a QA job to anyone struggling to "get a foot in the door" but do not rest on it. You're only as good as your skills, not your hours worked. If you want to use it like that you have to sharpen other skills, get yourself in the right corner of the room near coders and pick up some useful tidbits.
Equally if it's a career in QA you want, let higher ups know. So many people were "totally leader quality mate" but didn't tell management. Some guys did, and funnily enoough, they got the jobs by being forward and go getting.
Most of all: It can be a slog, but that light doesn't get any closer unless you start running for it.
0Sign inorRegisterto rate and reply
Nick Parker Consultant 8 years ago
QA can make or break a launch window and so they should. During my time at SCEE, we saw a number of lazy developers/publishers who would attempt to slip a title into a launch window without completing their own internal QA thereby using Liverpool as an outsourced resource. Another thing I remember about the Liverpool team was the amount they could drink when they descended on London for office parties!
0Sign inorRegisterto rate and reply
Gareth Bourn Level Designer + QA Technician, Hello Games8 years ago
I think that anyone that has lived the truth of this article is more than ready to agree. I found my way into Level Design via being a tester so I am one of those people. I had some of the best times of my life and met fantastic people as well! Everyone fails to mention the people and the environment, you get a sense of brotherhood you genuinely do lose in other departments (which is why I decided to go Indie - so that I can get that feeling back!). However, I do believe to become an Artist or a Coder, realistically, you will need a wealth of knowledge/talent that starting in QA without an education will not offer you (not to take away any shine from my own or any other role). I can build levels that play well from QA experience but I certainly can't draw and then model a sweet looking dragon or code even the most basic of functions!
0Sign inorRegisterto rate and reply
A good QA person can be a HUGE asset to any software developer. I never understood why they arent valued more. Good QA people have an instinct that cant be really taught, they understand the product, the goals, the mindset of the developers, they understand the logic even if they may not be able to directly code themselves.. As a designer/coder you are doing yourself a huge disservice if you dont have a valued QA tester. When you find a good one, why promote him out of that job? Give him a raise, Dont insist that QA is some low level job, its not, its vital.
I sometimes relate it to a golfer and his/her caddy. If you have a great caddy, you dont say, your great at this and helping us tremendously, now go do something else, makes little sense.

This industries view of QA personnel goes a long way in explaining why so many half assed and buggy games get shipped.

Edited 5 times. Last edit by Todd Weidner on 29th April 2013 6:06pm

2Sign inorRegisterto rate and reply
Bobby Wertheim Publishing Manager, SEGA Europe8 years ago
SCEE FPQA was my "introduction" to the industry, and I can confirm QA can be a good stepping stone to get to where you want to get to in the industry (for all the reasons mentioned already). There are good basic skills and knowledge you can pick up by being in QA, and this can be applied to your future role whether you decide to stay within QA or not. Just remember to stay focused, know what you want and persist in chasing that dream! They say patience is the key to virtue, but be proactive. There is no point spending your working life being unhappy... whatever your current job maybe :) That is all.
1Sign inorRegisterto rate and reply
Private Industry 8 years ago
Interesting read. From my experience it can be a lot easier to get into other areas like design or programming at smaller studios. When I started I was working as a functional tester, 1st party requirements tester, content creation, and AI tweaking at a small developer for console games. Because of the limited amount of resources it can happen quickly that you pick up other work as well with the right dedication.

The bigger companies get the more specialist it gets so I can see why they got different jobs after they moved away from SCEE. There are also a lot of positions you can move into more naturally from QA like certain manager positions or Producer ( seen plenty of people that went from QA to being a Producer) or just in general higher QA positions.

The only thing about the article thats a bit off putting is phrasing of QA abeing seen as minions or workhorses. I neverworked anywhere where QA people have been viewed this way and when there is overtime designers, programmers and so on work as well. It's not like they go home and only the QA department stays.

Edited 1 times. Last edit by Private on 29th April 2013 7:48pm

0Sign inorRegisterto rate and reply
Tom Sloper Executive Producer, Sloperama Productions8 years ago
This article backs up what I've been saying for years, and adds a lot of important insights. I've just made this required reading for my QA students.
1Sign inorRegisterto rate and reply
Andy Samson QA Supervisor, Digital Media Exchange8 years ago
This is truly inspirational and an eye opener. A lot of people who couldn't last working as QA find it mundane and tedious because they lack passion and the eagerness to continuously learn new things. So far I find it a lot more enjoyable than when I was a Game Master, instead of just playing the game and hosting events for other players I am actually working as an essential part of game development.

I learned a lot of great things when I got access to game design as well as assets creation. When I became supervisor I occasionally had the chance to submit some of my own proposals for fixing usability issues, enhancing gameplay and creating new features. What's even better is when they actually liked them and you see your concepts come to life in the virtual world.
1Sign inorRegisterto rate and reply
Nick Burcombe CEO & Co Founder, Playrise Digital Ltd.8 years ago
I know QA is very difficult for small developers like ourselves to test in any serious volume, but I wish we'd had 2-3 very good testers available to test Table Top Racing. It's an essential role and good QA people actually save time and money. I know that when you're working with the best QA guys - the ones who know enough about the dev process to really communicate what's going on, they are an absolute blessing. Anyone in QA wishing to advance their career, my advice would be to take every opportunity you can with the producer or the teams to ask questions and gain knowledge as you go along. It takes time. Game dev can be a bit of a mystery when you're not on the team, but gathering information will go a long way if you're persistent. I'd encourage devs to try - if there's time - to respond to QA guys with a little more insight and information where possible. That's how it worked for me back in the Psygnosis days - Tester No. 1 (1989 - yikes). I spent A LOT of time actually sitting with developers in those final stages. Only by asking the questions do you gain useful knowledge. QA Managers/Supervisors - spot your stars - the ones with a real passion for the business - and try and foster good relationships between QA and Dev Teams. Maybe try and organise some time for those with the right attitudes to be in with the team for a while (possibly nearer the start of the QA process). For the QA staff with a professional attitude, trustworthiness (I know there are security issues) and a genuine interest in trying to get further themselves in the industry, I think this could work well - for both parties.
1Sign inorRegisterto rate and reply
Jesse Penning Content manager, Ganz8 years ago
I love reading these comments. It shows that, though in pay, QA is behind, in spirit, it is ahead. Nobody forgets the QA struggle.
0Sign inorRegisterto rate and reply
Richard Westmoreland Senior Game Designer, Codemasters Birmingham8 years ago
QA staff are the hardest working and often least appreciated and paid members of our industry. Many companies don't appreciated their QA staff at all, but a good QA Tester can save a company so much time and money. QA was also my foot in the door and I really did get a lot of useful experience doing it.

I've also seen the other side, where QA staff are literally anyone who will take the job. They will then get exploited until they quit, at which point they are replaced by anyone else naive enough to take the job. When this happens the projects often suffer.
0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.