So you think video game research sucks?

It's common for an industry analyst to be challenged on findings and methodology. It is the nature of the gig. And in all fairness, games researchers seldom speak about what goes into the proverbial sausage: instead they often shroud themselves in mystery and offer only obfuscating explanations about how they know what they know. But I have to admit that I was hurt by Rob Fahey's recent claim that in describing the games market, "models and figures are pulled out of analyst's backsides with wild abandon." He paints a picture of market research as a discipline where "an infant let loose with coloured crayons and some graph paper would probably have a similar margin of error to their data."

Fortunately, Mr. Fahey is wrong. Here's why.

Let me start by stating that I am the founder and CEO of a games market research company. While completing my doctorate degree on the topic of video games, I ended up working for several game consultancies and analyst firms, and soon after graduation got frustrated by the lack of reliable research on digital games. Five years later, my team and I work with both major publishers and financial firms to identify major trends and quantify changes in the digital games industry. Secondly, here I will only speak for my own firm. Other companies out there can speak for themselves and I have no interest in muddying the public understanding of their preferred methods.

Under-informed and narrow

Mr. Fahey summarizes his argument by stating that "most of the sector analysis and research conducted on this industry is awful. It's under-informed, narrow and rarely exposes its methodology well enough to understand and account for its flaws."

Certainly there are many flaws in the practice of games research. Describing a global entertainment market that consists of a wide spectrum of different consumer practices and preferences, and which is often dominated by only a handful of secretive companies is, indeed, challenging. Perhaps, more so than anyone else, we know that we don't know and can't know certain things.

When we release information publicly we do so for two reasons. First, we obviously want to draw attention to what we do and what we have for sale. Research in any field is a dialogue rather than the dry transfer of information, and releasing recurrent updates and insights is our way of contributing to a larger industry conversation. Second, we release data into the ether because we're looking for feedback. When we say that League of Legends generates a billion in revenues, we are quite literally testing our knowledge against that of an entire market. There's nothing more terrifying than having a publisher strike back at your numbers, but it's the quickest way to find out if you're wrong.

"We, as market researchers, are just as skeptical, if not more so, than anyone else. We've heard it all. We've seen entire industry segments come and go"

So you must understand that merely stating that our research is flawed only serves to silence what should be an open conversation about how we define and measure the industry in which we all work. If the headlines that carry our numbers appear shallow, rather than blaming analysts, look to how media outlets operate. There are few truly discerning news outlets in the games industry today that actually go through the trouble of fact-checking, following up, and insisting on an objective, rational tone when reporting on findings. More than I care to admit, I have had to reach out to some aspiring games writer who managed to twist a terse headline into something entirely different and incorrect. I'm sure you can imagine that with a name like Joost van Dreunen I've seen a myriad of misspelled versions over the years. The same people that criticize the accuracy of our work struggle to spell my name.

Trust as currency

It is a beautifully corporate phrase, isn't it? As much as one may think it's all about pushing a bunch of numbers around, research is a network-based business. In fact, most of our customers come to us through referrals or, at the very least, will call up a friend at another company to validate us.

Our research team spends a fair share of time attending conferences and visiting customers to understand what their problems are and how we can help solve them. People won't tell you over email what keeps them up at night, so we go visit them in person. You see, as a research firm that specializes and exclusively deals with the games industry, we built our company slowly, convincing publishers and developers one-by-one. Over time, this trust has afforded us the insight into not just what goes on in one or two studios, but in most companies. And that's part of why people pay us: to give them a sense of what others are up to, how they deal with problems, how to identify new markets, and so on.

Transparency

Precisely because market research lacks poetry (necessarily so!), it creates most of its value in utility. To that end, we are obligated to constantly explain ourselves. You'd be surprised as to the extent of arrogance we encounter at all levels of the games industry, where egos drown out logic ("I have no evidence but I'm pretty sure your data is incorrect") and personal politics play into major decisions.

But in the spirit of transparency, here's how we do it: using the monthly transaction data we receive from developers, publishers and payment providers totaling a data set of 37 million spending online gamers we (1) develop a base model. The extent of our data gives us insight into seasonal changes, genre performance and major variables. To establish anchor points we then (2) compile and integrate publicly available information from a range of sources, including quarterly filings, press releases, marketing materials, executive interviews, conference presentations and company visits. To ensure we didn't miss anything, we (3) cross-reference our initial findings with additional qualitative research, such as recurrent survey studies. This allows us to gut-check our numbers and provides context such as wallet share, device preferences, and so on. This ultimately gives us (4) title-level estimates expressed in monthly active users, conversion rates and average spend, which we (5) continuously update and re-validate and, finally, (6) update on a monthly basis as new information becomes available through our partner network.

If that felt too long to read, congratulations, you now know why researchers provide executive summaries: because most people don't have the time to read through all the work that goes into arriving at an accurate description of the market and its trends. Add to that the five years or so it took us to broker the relationships necessary to gain access to spending data on mobile and social games, Steam, shooter games-- totaling over 500 titles and 26 markets-- and you get an idea of what's involved. Seriously though, if you can help us improve on this model, we're all ears.

A healthy dose of skepticism

What Mr. Fahey seems to forget is that we, as market researchers, are just as skeptical, if not more so, than anyone else. We've heard it all. We've seen entire industry segments come and go. We've been invited to analyst meetings where all the latest and greatest is showcased and it is our duty to test what we're told against what we know.

But what is truly irksome is the way in which solution providers tend to present their own, internal data as market averages. Think of all those traffic providers claiming that there are brilliant ways to avoid growing user acquisition costs. Or that endless stream of infographics compiled of random information pulled from the internet, made to look informative by dolling it up. This is what Plato rebelled against, and this is where the way in which 'data' is presented confuses. You may think us infantile, but I assure you we researchers respond with a healthy dose of adult rage when it comes to the general misinformation that clutters the conversation. In our club, the first rule of working with data is to be suspicious at all times.

What have you done for me lately?

One thing that we provide, again and again, is the rationalization of what is otherwise a rollercoaster of subjective activity known as the creative process. We deal with the dreamers and geniuses that make possible the interactive fantasies that make up this industry. In my experience, people in this industry pour their heart and soul into their work. It is what drew me to this business in the first place, and it is what keeps me motivated. I resent the notion that our efforts are exclusively focused on maximizing revenues. Or, better yet, I've worked with enough game companies by now to know that is not how this industry works.

So, Mr. Fahey, consider this an open invitation to learn more about our methodology and our research team. I'm sure you'll find them more capable than infants, much like your writing exceeds that of a room full of monkeys with a typewriter.

You know where to find me. Just Google it.