Sections

"Are we all just fooling ourselves to some degree?"

A look at the limits and logistics that shape market research

Let's talk about market research. We cover a lot of it at GamesIndustry.biz, and like any media content you consume, it goes well with a grain of salt.

In many cases, the research can be boiled down to a few simple ideas. People are going to spend lots of money on virtual reality in the next five years. People are going to spend lots of money on eSports in the next five years. Emerging markets are going to spend lots of money on games (perhaps even virtual reality and eSports) in the next five years.

I'm pretty confident that all of these things are true. Anyone who's been paying attention to the industry likely feels the same way. We should all be able to agree that these things are very likely going to happen; we're just going to disagree on exactly how much money we're talking about, and whether or not that number would constitute failure. And we may disagree wildly on those issues because we're all measuring performance against our own differing expectations, and we're talking about a future with an incredible number of variables at play.

Will VR find its killer app? Will motion sickness give the tech a bad reputation, or will a high cost of entry keep people away? Will eSports evolve into a wide field supporting a plethora of games, or will it be League of Legends, Starcraft II, and Counter-Strike: Global Offensive for the next 15 years? Will efforts to televise competitions break through to mainstream audiences? Will the Latin American market settle on a frictionless payment standard for mobile users? Will the hassles and costs of doing business in China finally cool enthusiasm for the massive potential audience it offers?

"It's not a popular thing to say that things aren't growing. There's this emphasis on the economics of permanent growth. And that's a mindset. That's what people want to hear."

Joost van Dreunen

Anyone can guess at the answers, and they may even have reasonably sound justifications behind them, but they're still just guesses. Oftentimes, they're guesses that analysts at market research firms are making with limited information. People already deeply entrenched in virtual reality development, eSports organizations, or emerging markets probably understand their fields better than industry generalists trying to quickly bring themselves up to speed on complex niches to make the best guesses they can on their latest assignment.

Regardless, people are paying for those guesses, often to the tune of thousands of dollars per report. It sounds like an exorbitant amount, but when you have an industry where people need to justify every action with numbers and often have a research budget to help them obtain those numbers, market research suddenly becomes a very viable field.

The catch here is that people looking to justify a course of action typically have a course of action in mind. So if you were a start-up looking to attract venture capital for your VR eSports project, would you be buying the research that says, "This stuff is a fad," or would you be buying the rosiest research you could find, the one that tells prospective VCs your idea is likely to make a sickening amount of money?

"It's not a popular thing to say that things aren't growing," Superdata Research CEO Joost van Dreunen told me when asked about the subject. "There's this emphasis on the economics of permanent growth. And that's a mindset. That's what people want to hear. You want to make your boss happy by saying, 'This is amazing. Look at this cool thing I found. Give me a budget and I'll build this because we have all these reports that say it'll be fantastic.' That's a much more interesting thing for people to do than saying, 'We should be cautious.'"

Van Dreunen said that he received a number of angry emails from companies last year for suggesting that the mobile market is saturated.

"The moment you start saying, 'We need to watch out for this because it looks like the costs of production and marketing are going up tremendously the next 18 months,' now you're one of the suits and you're ruining all the fun," van Dreunen said, adding, "If somebody's paying you to write a report on how awesome something is, does that affect your findings? It's a tricky thing for researchers to walk that line."

Newzoo CEO Peter Warman also acknowledged the abundance of positive research in the field, but added that there was a lot of positive stuff to write about in gaming these days. And it's not necessarily that these research firms never met a trend they didn't love; there just might not be a market for research that questions the conventional wisdom.

"Everybody has actual data as sources, but it's a different situation from the past where 99 percent of the market was just counting the boxes times the prices. Over the past years, all figures have become estimates, you could say."

Peter Warman

Take VR, for example. Warman said that the technology will be huge "a generation from now," but Newzoo research is focused on software revenues, and its initial research into VR suggests there aren't tremendous opportunities in that field in the short term. And to hear him tell it, the reason that research hasn't been offered for sale is explicitly because it's not positive enough.

"We don't feel ready to report on potential game revenues because it's zero now, and we think it will actually remain in the millions or tens of millions for the coming year and a half or so, for software," Warman said. "We tell this story to press a lot when we get asked about VR, but we have not chosen to write down that opinion as an opinion piece, which would counterbalance the super positive articles a little bit. So I do agree that we should feel responsible to do that as well."

The issue is compounded by a lack of clarity in the industry, partly from the abundance of players in the market proclaiming their own set of numbers to anyone who will listen (both van Dreunen and Warman seemed particularly skeptical of analytics firms calling their numbers research a marketing tool), and partly from the ever-shifting business models and distribution methods in gaming.

"The world is completely digital and there are no actual box counts anymore," Warman said. "Everybody has actual data as sources, but it's a different situation from the past where 99 percent of the market was just counting the boxes times the prices. Over the past years, all figures have become estimates, you could say. And that has led to the problem, as you call it, that we see today. It's a lot more complex to [track] that in a sound way."

Even the lingo of the industry hurts. It's hard to make apples-to-apples comparisons when everyone is using different classifications for terms like 'social gaming,' 'casual gaming,' and 'MMOs.' Earlier this month, I even had one research company explain a discrepancy in its forecast numbers by explaining that it was including physical copies in the number given for 'digital games software sales.'

"This is somewhat the bane of my existence," van Dreunen said. "I'll present a number and somebody will say, 'Well I have [a different number]. How does it line up?' It's like, I don't know. I don't know their methodology. I don't know who worked on this. So everybody feels like they have access to information, but there's no real actual knowledge in this... It's really, really confusing."

"[E]verybody feels like they have access to information, but there's no real actual knowledge in this... It's really, really confusing."

Joost van Dreunen

As Warman said, that sort of confusion can be damaging, not just to the research companies involved, but to the industry itself. He said that several years ago when NPD dominated the market of games reporting and Superdata and Newzoo were just starting up, he and van Dreunen discussed the NPD's attempts to report on digital sales and the ways they could improve on them.

"When I think back to my discussions with Joost from Superdata, one of our objectives was to make sure that our high-level data would not be... Well, we would sort of check with each other. Because it doesn't help the market if global figures or US figures in total are very different. It confuses people and it doesn't help anyone."

As an example, he brought up eSports as one sector where Newzoo and Superdata came to different conclusions. In May, Superdata released a free market brief putting 2015's worldwide eSports revenues at $612 million. The next month, Newzoo released its Global Growth of eSports, which only projected a $250 million worldwide market in 2015 and cost $4,900.

"That made us really mad," Warman said. "There was no product behind it; it extrapolated $200 million revenue in Korea from 2007 to the current day, completely disregarding the crash that happened in Korea. That was an example where it really hurt us. We had to explain a lot. I hope they had to explain a lot. And it showed how confusing it can be for a market if the high-level numbers are so different."

So not only is a confirmation bias involved in research, there's also an incentive to release reports that essentially echo each other. If your findings are too negative, there might not be a market for them. If your findings are different from other companies' findings, expect to get some backlash.

And that doesn't even begin to get into the challenges inherent to studying an industry that changes as quickly as games. There's a lag between the time a trend starts picking up steam and the time it takes researchers to identify it as a viable subject for a report, collect data on it, analyze the data, articulate their findings, and finally offer them to the world. And then customers need to get the report, evaluate the information inside, and adjust their own plans in line with the report's findings. Even if the report is fundamentally sound, the information within might be considerably less useful when it's actually put into action than it was when it was collected.

"It's too strong to call it a lie, but you join the party to some degree on your own terms. Ultimately, everybody to some degree is sipping from the Kool-Aid because you don't want to be the one that rains on the parade."

Joost van Dreunen

The crazy thing here is that these shortcomings are well understood, but almost everybody benefits from the system the way it is currently set up. The press gets some attention-grabbing headlines. The industry as a whole benefits from positive buzz. People wanting to explore these new fields are given a way to justify spending money to do so. The people who control the purse strings (and often need to invest in something) get reassurance that those enthusiastic explorers are not completely off the mark. And even the investors whose money is ultimately being spent are investing in a field people are confident has loads of growth potential.

"It's sort of an ecosystem on its own," Warman said. "It maintains itself."

Van Dreunen expanded a bit more on the idea, saying, "It's too strong to call it a lie, but you join the party to some degree on your own terms. Ultimately, everybody to some degree is sipping from the Kool-Aid because you don't want to be the one that rains on the parade. You don't want to miss out, and I think that's a big driver. Ultimately, you're dealing with humans. There's a rational way of approaching markets, industries, and opportunities and a methodology behind it, but then there's the sort of sticky, weird, human factor that just doesn't want to miss out on an opportunity, or is in a hurry to leave and rush to the next thing because that's where growth is going to be, and that's what's going to solve all our problems as a company, or whatever. People have very irrational reasons to make decisions, and once that catches momentum...

"People were talking about VR, but once a company like Facebook put down $2 billion, all of a sudden everybody's an expert and everybody's totally pro-VR. I never heard about it from all these people in my life, but all of a sudden after Facebook bought in, now it's the best thing after sliced bread. So you kind of have to be skeptical about that. Are we all just fooling ourselves to some degree?"

None of this is to dismiss the value of research out of hand or suggest that they are working backward from positive conclusions first and only later figuring out how to support them. But people need to exercise media literacy about these reports the same way they would the rest of the news. Is the methodology based on straight-forward point-of-sale data, or numbers extrapolated through a complex formula that the firm doesn't want to detail? Is the research based on a survey of 10 people, or 1,000? Do the people involved really know the field they're writing about? Do they need to? Is the research citing hard data from a game's past performance, or projecting industry-wide trends on a worldwide basis five years into the future?

Because without that critical evaluation and context, we're all just huddled around a Ouija board, asking unseen forces to show us the way, each one swearing that we aren't the ones nudging the planchette to the answer we all want to hear anyway.

Latest comments (8)

James Ingrams Writer 2 years ago
The money behind research means no data in survey is truly trustworthy. For 15 years the video games market, in my opinion, has become borderline corrupt. One example, however big a movie release (Star Wars, etc) it can still get slammed my the movie reviewers if it's bad. Does anyone really think Fallout 4 is not going to get 90%+ review scores however lacking it is?

The media is there to support the industry, when it used to be there to support the customer and hold the industry's feet to the fire if necessary. No games review website, for example, should accept invitations to games publishers, or accept free copies of games from them. To many reviews are based on beta copy, or by getting many games free we have a media that gives low scores for lack o0f innovation when gamers are much more conservative and won't want enormous innovation!

I trust the video games industry and technology industry in general as far as I could throw it! This is why I believe we are going to see continuing decline in the profitability of AA gaming, the levelling off of the indie sector. And the general failure of expensive VR due to the future lack of disposable income due to the ongoing global economic problems.
0Sign inorRegisterto rate and reply
Connor Martin Aspiring game designer/tester 2 years ago
VR will be an experience platform, not many games I could at least theorise would interest me enough to pay for it, I already stand at the waters edge on the "next" gen of consoles apart from a Wii U for the delicious exclusives when it churns one out. Granted a joy of a new means of providing entertainment is finding things I never knew I wanted till I saw it but still......best of luck, honest to god its gonna be hell out there.
0Sign inorRegisterto rate and reply
Klaus Preisinger Freelance Writing 2 years ago
iOS, Android, Steam, Twitch, Youtube, etc.

All of those platforms rose to incredible heights over the years. The one thing they have in common is that neither of them required a killer app. None of them needed a Jesus Christ of consumer goods to redeem them from bad sales. I hope the same can be said for VR down the line. That there is not that one killer app that is the almighty catalyst of converting people into consumers, but instead a solid stream of content that will win over people one by one. That is why everybody has a TV and still watches movies and TV shows on it. Because it is not about the 99% of god-awful programming, it is all about the 1% people like and that one percent is different for most.
1Sign inorRegisterto rate and reply
Show all comments (8)
Curt Sampson Sofware Developer 2 years ago
Connor, if I'm reading you right, you feel that VR isn't really going to make a lot of difference to existing games. Can you tell me why you think so? It seems to me that it's got a reasonable chance of being a massive leap in controlability for many styles of games, probably as big a leap as when twin joysticks were added to the original Playstation controller.

If I'm right, the main issue is simply going to be the price tag. (And this gives Sony a huge advantage, if Occulus is really right about their requirement for a $400+ graphics card.) That puts a definite crimp on things, but in the end that will simply be a delay of several years, rather than a permanent problem.

Edited 1 times. Last edit by Curt Sampson on 23rd October 2015 3:26am

0Sign inorRegisterto rate and reply
Nick Parker Consultant 2 years ago
Wow, how we got off the subject of research! You need people passionate about discovering the facts, mining trusted sources on a regular basis and only interested in real data not optimistic over reporting. I've been analysing the games market for 23 years and have market data going back to the launch of Game Boy in the late 80's for most markets. I now source from local trusted agencies for digital platforms either directly from the console manufacturers and PC platforms or by searching the net for publisher/developer CEO sound bites, company reports and bonafide original data suppliers. I pay scant regard to headlines about the over valuation of the market and long term forecasts. Stick to the facts even if they are not exciting and don't support your business plans.

Oh, by the way, by triangulating a number of sources, the global VR headset market could be 3 million installed base by the end of 2016 and ten times that by 2020.

Edited 2 times. Last edit by Nick Parker on 23rd October 2015 2:28pm

1Sign inorRegisterto rate and reply
Paul Johnson Managing Director / Lead code monkey, Rubicon Development2 years ago
Are we all just fooling ourselves to some degree?
No WE are not. I've been predicting a fizzle since the beginning. The mass market just won't want this stuff bolted to their face. It's why it failed the first time and nothing's changed since on that front.

Edited 1 times. Last edit by Paul Johnson on 23rd October 2015 5:25pm

1Sign inorRegisterto rate and reply
Curt Sampson Sofware Developer 2 years ago
The mass market just won't want this stuff bolted to their face. It's why it failed the first time and nothing's changed since on that front.
So you think the lack of good motion tracking and graphics power to translate that motion into something that looks right had nothing to do with it?

I'm certainly not going to predict a guaranteed huge success for VR this time 'round, but the technology does this time seem to me to have a couple of critical things that were missing (or, alternatively, "utterly broken") in previous headsets.

The way I see it, a VR headset is not primarily a display, it's primarily a controller. And none of the ones up to this point have been even half-way decent controllers.
0Sign inorRegisterto rate and reply
Paul Johnson Managing Director / Lead code monkey, Rubicon Development2 years ago
So you think the lack of good motion tracking and graphics power to translate that motion into something that looks right had nothing to do with it?
The first pass of this, it was too heavy. We all got to that sentiment long before reaching the "it's actually pretty crap and claustrophobic anyway" follow up. In fact virtuality came to us at ATD early for support and my then bosses punted it, for which us minions were grateful on health grounds!

It's better now and I actually do think this stuff will find a market, but a very small one. It's certainly not going to be the ushering in of a new dawn like those who want it to be will have us believe, and I must admit to finding all this blind support a bit naieve tbh. A triumph of desire over practicality, like MJF's actual self-lacing shoes.

I'm kinda reminscent of Kinnect here, although they're obviously different things. It started out all "OMG you don't need to hold anything?!??" and then a while later it was more "I'll pay you to take this away".

Edited 1 times. Last edit by Paul Johnson on 24th October 2015 8:35am

2Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.