Sections

So you think video game research sucks?

Superdata's Joost van Dreunen explains the method to a gaming analyst's madness

It's common for an industry analyst to be challenged on findings and methodology. It is the nature of the gig. And in all fairness, games researchers seldom speak about what goes into the proverbial sausage: instead they often shroud themselves in mystery and offer only obfuscating explanations about how they know what they know. But I have to admit that I was hurt by Rob Fahey's recent claim that in describing the games market, "models and figures are pulled out of analyst's backsides with wild abandon." He paints a picture of market research as a discipline where "an infant let loose with coloured crayons and some graph paper would probably have a similar margin of error to their data."

Fortunately, Mr. Fahey is wrong. Here's why.

Let me start by stating that I am the founder and CEO of a games market research company. While completing my doctorate degree on the topic of video games, I ended up working for several game consultancies and analyst firms, and soon after graduation got frustrated by the lack of reliable research on digital games. Five years later, my team and I work with both major publishers and financial firms to identify major trends and quantify changes in the digital games industry. Secondly, here I will only speak for my own firm. Other companies out there can speak for themselves and I have no interest in muddying the public understanding of their preferred methods.

Under-informed and narrow

Mr. Fahey summarizes his argument by stating that "most of the sector analysis and research conducted on this industry is awful. It's under-informed, narrow and rarely exposes its methodology well enough to understand and account for its flaws."

Certainly there are many flaws in the practice of games research. Describing a global entertainment market that consists of a wide spectrum of different consumer practices and preferences, and which is often dominated by only a handful of secretive companies is, indeed, challenging. Perhaps, more so than anyone else, we know that we don't know and can't know certain things.

When we release information publicly we do so for two reasons. First, we obviously want to draw attention to what we do and what we have for sale. Research in any field is a dialogue rather than the dry transfer of information, and releasing recurrent updates and insights is our way of contributing to a larger industry conversation. Second, we release data into the ether because we're looking for feedback. When we say that League of Legends generates a billion in revenues, we are quite literally testing our knowledge against that of an entire market. There's nothing more terrifying than having a publisher strike back at your numbers, but it's the quickest way to find out if you're wrong.

"We, as market researchers, are just as skeptical, if not more so, than anyone else. We've heard it all. We've seen entire industry segments come and go"

So you must understand that merely stating that our research is flawed only serves to silence what should be an open conversation about how we define and measure the industry in which we all work. If the headlines that carry our numbers appear shallow, rather than blaming analysts, look to how media outlets operate. There are few truly discerning news outlets in the games industry today that actually go through the trouble of fact-checking, following up, and insisting on an objective, rational tone when reporting on findings. More than I care to admit, I have had to reach out to some aspiring games writer who managed to twist a terse headline into something entirely different and incorrect. I'm sure you can imagine that with a name like Joost van Dreunen I've seen a myriad of misspelled versions over the years. The same people that criticize the accuracy of our work struggle to spell my name.

Trust as currency

It is a beautifully corporate phrase, isn't it? As much as one may think it's all about pushing a bunch of numbers around, research is a network-based business. In fact, most of our customers come to us through referrals or, at the very least, will call up a friend at another company to validate us.

Our research team spends a fair share of time attending conferences and visiting customers to understand what their problems are and how we can help solve them. People won't tell you over email what keeps them up at night, so we go visit them in person. You see, as a research firm that specializes and exclusively deals with the games industry, we built our company slowly, convincing publishers and developers one-by-one. Over time, this trust has afforded us the insight into not just what goes on in one or two studios, but in most companies. And that's part of why people pay us: to give them a sense of what others are up to, how they deal with problems, how to identify new markets, and so on.

Transparency

Precisely because market research lacks poetry (necessarily so!), it creates most of its value in utility. To that end, we are obligated to constantly explain ourselves. You'd be surprised as to the extent of arrogance we encounter at all levels of the games industry, where egos drown out logic ("I have no evidence but I'm pretty sure your data is incorrect") and personal politics play into major decisions.

But in the spirit of transparency, here's how we do it: using the monthly transaction data we receive from developers, publishers and payment providers totaling a data set of 37 million spending online gamers we (1) develop a base model. The extent of our data gives us insight into seasonal changes, genre performance and major variables. To establish anchor points we then (2) compile and integrate publicly available information from a range of sources, including quarterly filings, press releases, marketing materials, executive interviews, conference presentations and company visits. To ensure we didn't miss anything, we (3) cross-reference our initial findings with additional qualitative research, such as recurrent survey studies. This allows us to gut-check our numbers and provides context such as wallet share, device preferences, and so on. This ultimately gives us (4) title-level estimates expressed in monthly active users, conversion rates and average spend, which we (5) continuously update and re-validate and, finally, (6) update on a monthly basis as new information becomes available through our partner network.

If that felt too long to read, congratulations, you now know why researchers provide executive summaries: because most people don't have the time to read through all the work that goes into arriving at an accurate description of the market and its trends. Add to that the five years or so it took us to broker the relationships necessary to gain access to spending data on mobile and social games, Steam, shooter games-- totaling over 500 titles and 26 markets-- and you get an idea of what's involved. Seriously though, if you can help us improve on this model, we're all ears.

A healthy dose of skepticism

What Mr. Fahey seems to forget is that we, as market researchers, are just as skeptical, if not more so, than anyone else. We've heard it all. We've seen entire industry segments come and go. We've been invited to analyst meetings where all the latest and greatest is showcased and it is our duty to test what we're told against what we know.

But what is truly irksome is the way in which solution providers tend to present their own, internal data as market averages. Think of all those traffic providers claiming that there are brilliant ways to avoid growing user acquisition costs. Or that endless stream of infographics compiled of random information pulled from the internet, made to look informative by dolling it up. This is what Plato rebelled against, and this is where the way in which 'data' is presented confuses. You may think us infantile, but I assure you we researchers respond with a healthy dose of adult rage when it comes to the general misinformation that clutters the conversation. In our club, the first rule of working with data is to be suspicious at all times.

What have you done for me lately?

One thing that we provide, again and again, is the rationalization of what is otherwise a rollercoaster of subjective activity known as the creative process. We deal with the dreamers and geniuses that make possible the interactive fantasies that make up this industry. In my experience, people in this industry pour their heart and soul into their work. It is what drew me to this business in the first place, and it is what keeps me motivated. I resent the notion that our efforts are exclusively focused on maximizing revenues. Or, better yet, I've worked with enough game companies by now to know that is not how this industry works.

So, Mr. Fahey, consider this an open invitation to learn more about our methodology and our research team. I'm sure you'll find them more capable than infants, much like your writing exceeds that of a room full of monkeys with a typewriter.

You know where to find me. Just Google it.

Latest comments (3)

Steve Peterson Marketing Consultant 4 years ago
An excellent rebuttal, Joost. I'm skeptical of numbers that I see from many sources, knowing how difficult it is to verify so much of what's being said (especially by privately held companies, which have no legal obligation to disclose numbers and plenty of incentives to make themselves look better). It's good to see the effort being put in to crosscheck numbers and make your data as reliable as possible. Certainly your business in the long run depends on your numbers being as good as you can make them, so you have a clear incentive to keep working to improve your information.

All that said, I think you'd agree that the further out we see any predictions being made about revenues or sales, the greater the error bars become. When we see analysts talking about 2019, that's going to be far less reliable than 2015 predictions. There's just too many things that can occur to alter numbers over five years. Still, it's always useful to see where the trend lines would take the business if things don't change, and maybe work towards changing those trend lines.

I know how much harder it is to get numbers these days versus 20 years ago, when everything you needed to know was tracked by retail sales alone. The market is far more complex now, and we all need to have the best numbers we can get about the business to guide our decisions. I applaud your efforts, and those of other analysts, to develop methodologies and relationships to provide that desperately needed data.

Of course, I'll still be challenging you about where you get those numbers, especially when I find them surprising. Often knowing how numbers are obtained is as important as the numbers themselves.
0Sign inorRegisterto rate and reply
Brian Lewis Operations Manager, PlayNext4 years ago
Having been in the position of having to fact check some of the numbers presented by Superdata, I, personally believe that they are reasonably accurate. Much of what they present are estimates (because they don't always have all of the numbers) but they do follow the trends, and the end results are reasonably accurate. I have had some of the numbers that they have 'estimated' and when reviewing their results, have been pleasantly surprised about how accurate they are. Based on what I could verify, the data seems to be pretty accurate, and very thorough. I give them a LOT of credit for doing a tough job, and getting it right.
0Sign inorRegisterto rate and reply
Peter Warman CEO & Co Founder, Newzoo4 years ago
Joost, thanks for representing our field. I saw Robs article and atthattime decided to refrain from responding despite Newzoo being mentioned as example. With our companies we both are investing literally all our time and effort in understanding, analyzing and reporting on a market that is in the midst of a global revolution. That is tough and challenging, if not impossible. Tremendously challenging from a business and intellectual perspective.

We share the belief that no data is perfect and dare to state that publically. But we sure both work our behinds off to get as close as we can. And indeed, we cannot do this without feedback from our clients (which I believe include over 90% of all big game companies as well as most hardware and media giants) and others out there who operate commercial companies. At the same time, we certainly can do without the lazy and easy feedback from guys like Rob. Who is he to question data and insights that are validated, used and paid for by the most renowned game, hardware and media companies on the planet?

Luckily a growing share of journalists and editors takes the time to understand and report on methodology and requests more granularity if he or she believes their audience will question the large numbers. We also put out a lot of data and insights for free. Partially for the same reasons as you state but also to help the smaller developers. We have tried to sell lower priced services to assist the indies but it did not work out. Superdata and Newzoo together put out enough data and expertise to give them a start at least.

If Rob wants to go back to the good old NPD era when everything was sold in a box and Asia did not matter to US companies that is fine for me. Just watch out that you do not miss the next boat. Underestimating a market (as NPD did with mobile and all other digital business models for a couple years) can be extremely dangerous for companies that rely on their data. Just as Joost concluded, I also invite anyone who is interested to take a look at how granular the data is behind our recent $25bn mobile game announcement to jump on a call. You will be amazed how many datapoints and sources are combined to give us the confidence to publish this.

I am proud to be working at a research company comparable the one Joost heads up that has stepped up at a moment traditional forces were not willing or able to analyze change and take the risk of reporting their findings publically ........
0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.