Games will need a radical rethink to work in AR | This Week in Business

This Week in Business is our weekly recap column, a collection of stats and quotes from recent stories presented with a dash of opinion (sometimes more than a dash) and intended to shed light on various trends. Check back every Friday for a new entry.

I was combing through video game patents recently (as one does), and I came across the above picture of what appears to be a pirate and a monster pirate menacingly staring out of two different front windows on an otherwise unremarkable and nice-appearing home.

The patent was for a "virtualized environment" system of the sort that could be used to spruce up haunted houses. And while that's perhaps augmented reality-adjacent at best, my first thought upon seeing the image was, "Augmented reality gaming is doomed."

Okay, not my first thought. My first thought was, "This looks like something out of Tom Hanks' 1989 dark comedy The 'Burbs," quickly followed by, "I wonder if The 'Burbs would be better appreciated today than it was back then" and, "Is The 'Burbs streaming anywhere?" After that came an IMDB rabbit hole recap of director Joe Dante's one-of-a-kind career, which is admirably all over the map. Gremlins, Explorers, The Howling, Looney Tunes: Back in Action, Hawaii Five-0, Amazon Women on the Moon… it just keeps going.

But immediately after that, my first thought was, "Augmented reality gaming is doomed." Or to be more slightly more nuanced about it, "augmented reality as it applies to contemporary ideas of gaming is doomed."

In some ways this gets back to my initial reaction to Apple's Vision Pro announcement and how the company didn't feature games in any significant way. I initially took that as evidence Apple was largely disinterested in the medium of games, an observation for which there is ample historical evidence.

But upon seeing those menacing patent art pirates glaring out of mini-McMansion windows, I started to wonder if there was another reason behind Apple's decision, one more fundamental to a lot of traditional gaming and the idea of immersion.

In one sense, having games play out within our real-world environments sounds like it would be a boon for immersion. What could help convince you something is real more than seeing it in your world, interacting with things that actually are real?

When we talk about immersion in games, we're usually talking about engrossing escapism more than convincing illusion

But when we talk about immersion in games, we're usually talking about engrossing escapism more than convincing illusion.

Video games are often power fantasies, and they place a premium on things that are, for lack of a better word, cool.

Lots of people find pirates cool. Monstrous or undead pirates? Even cooler. Space marines in power armor, ninja warriors, hooded assassins, scary weirdos in From Software games? All cool, and all build the kind of "player fantasy" on which AAA publishers have built their businesses.

But when you take those cool things out of the stories and environments that exist to reinforce their coolness, and you instead put them in our contemporary everyday world, they are going to stick out like a sore thumb and work against the immersion that one might expect to be a key benefit of using AR for games in the first place.

If you just need players to be completely captivated by the experience, it seems like VR would be the better option, entirely replacing their perception of the real world with a digital one. Traditional screens also work for that kind of immersion as players can focus on the world in that screen, but AR's mixture of the two threatens to constantly reinforce the difference.

It's like playing a VR game, but with a picture-in-picture camera feed in the corner of the screen showing you what you actually look like as you dodge enemy fire or pretend to climb a mountain wearing a giant headset in your living room.

So many video games work because they present a heightened fantasy of coolness for people to inhabit for a while, but AR by its nature seeks to juxtapose that fantasy with a mundane reality many users would like to escape.

That's how you end up with Kratos standing in your kitchen, covered in the blood of capricious and overconfident gods, right next to the cookie jar that will never stop smelling of Nutter Butters no matter how many times you wash it out.

I readily admit kitchen Kratos would be a cool thing to experience as a novelty, just like Microsoft's Hololens Minecraft and Magic Leap's whale splashing down at a school assembly. If they worked as advertised, it would be a whizbang bit of technology to make a fictional thing appear within a very real environment. But once that novelty wears off, I'm going to need someone to explain to me how these things make the actual game better.

Now obviously this possible ludo-corporeal dissonance isn't a fatal flaw, because some brands can work really well in an AR/real world setting.

STAT | $5.5 billion – Sensor Tower's lifetime estimate for Pokemon Go revenues as of November of 2021.

Others maybe not so much.

STAT | $39.4 million – Sensor Tower's lifetime estimate for Harry Potter: Wizards Unite revenues as of the same date, when Niantic announced it was shutting down the AR game after two years.

Regardless of which specific brands work and which don't, we're talking about a significant limitation to the possible range of experiences that could work in AR compared to what we already know can work on traditional screens.

Creating with one hand tied behind your back can actually be healthy, as working around limitations can force you to be creative in other ways. But making games in AR may be more like creating with both hands tied behind your back.

Even if we were to assume the technology is effective, unobtrusive, and inexpensive, AR poses similarly difficult questions from a design standpoint

Even if we were to assume that all brands could work equally well in AR and that the technology will be effective, unobtrusive, and inexpensive – and those are four entirely separate non-trivial problems – AR poses similarly difficult questions from a design standpoint.

If AR gaming is going to be anything grander than a new way to play games entirely possible in other formats, then the games need to understand and interact with the player's environment. And that only makes sense if the environment where you play – the specific arrangement of furniture and things around you – is meaningful to gameplay.

If the environment significantly impacts the gameplay, that suggests people are going to be having wildly varying experiences depending on the size and relative clutter of the place they happen to be. Will someone be able to play in the middle of an open field? How about on a crowded bus? In a bathroom stall? The more the game relies on the geography of one's immediate surroundings, the harder a design problem it becomes to avoid possible environments that would be sub-optimal or completely unviable.

The more designers control for that potential to ensure that every environment can still work, the more abstracted the impact of the environment becomes. For example, you could treat data from the environment as a randomized seed for a level generator, but at that point I wonder how the experience is in any way superior to what you could get using VR or a standard monitor.

One "solution" is to have the environment become meaningless wallpaper, as in Nintendo's first foray into augmented reality, the pre-installed 3DS game Face Raiders. It's a fun novelty and the AR aspect is attention-getting, but it is tangential to the actual game.

This is a common theme among AR games to date. Pokémon Go is held up as the gold standard of AR games, but it's the geolocation element and the Pokémon brand that make the game work, much more so than the AR. (It's telling that one of the first things beginner's guides for the game suggest is to disable the AR functionality.)

When we talk about "AR functionality," it seems we're mostly talking about what can be done with a camera and motion-sensing tech. Between headsets and depth-sensing cameras and gesture-recognition, there are a bunch of different ways that could be made better or worse, but those are the key elements as I see them.

And between the Wii, the Kinect, and the iPhone, we've seen no shortage of talented developers do their best to take cameras and motion-sensing tech and make fun games with them. Some have even succeeded, but the enthusiasm for those features has largely died down, even if they're all still around and contribute positively to games when used in an intelligent manner. Putting the cameras and the motion-sensing into a headset is an interesting change, but I don't see how it necessarily makes these same design fields suddenly more fruitful for developers.

Much of what we've seen in AR to date has run into similar problems where they either required player environments be arranged a certain way – defeating the point of AR games that are aware of and adapt to your environment by making you conform your environment to the game's specifications – or they made the environment essentially meaningless to the gameplay, in which case the game could be done just as well without using the tech.

As for what has held actual promise, the most interesting AR experiment I've seen so far is Mario Kart Live, which added some video game elements to the real-world activity of RC car racing. That was never going to be a breakthrough killer app for the idea of AR – it was still limited by the reliance on pricey physical elements and the need to clear out a 10x12 ft chunk of space for a course – but I believe it points the way to a viable approach to AR gaming that can work, even if it won't eclipse the traditional gaming market in revenue anytime soon.

I also think there's potential in AR games that eschew some of the qualities we often take for granted about games: goals, challenges, objectives, and so on. I'd love to see games that could work with one's neighborhood or home to create a sort of found object art, for instance. But I don't expect anything of the sort to justify the AR hype we've seen ebb and flow for a decade now.

In short, I don't think companies will find success converting games into AR experiences. This won't be like the late 2000s when we were told mobile was the future for a very long time and then touchscreen smartphones made a breakthrough and suddenly that promised future was very much the present.

And I freely admit this may just be a failure of imagination on my part. There's a reason I'm a games journalist and not a fiction writer or a game developer, after all.

But if AR is going to find a viable use case in games, I believe it will take a major rethink of how games work and what experiences they offer.

It will not happen by taking things that already work in games and putting them in the real world. Instead, I believe AR's potential lies in taking things that already work in the real world and giving them a veneer of video games.

Much like the gamification trend that took hold around the same time we started hearing about augmented reality, AR represents less an opportunity for the games industry than an opportunity for others to apply the concepts and techniques of games to their own fields.

The rest of the week in review

STAT | 65% - The percentage of votes against an Activision Blizzard shareholder proposal to adopt a union non-interference policy that was the subject of last week's column.

QUOTE | "Xbox has lost the console wars, and its rivals are positioned to continue to dominate, including by leveraging exclusive content. Xbox has consistently ranked third in consoles behind PlayStation and Nintendo." – As spotted by The Verge, Microsoft's lawyers waved the white flag in a filing ahead of this week's hearing on the FTC's objection to the Activision Blizzard merger.

The hearings will continue into next week, and you can get a recap of some of the more scintillating scuttlebutt here.

QUOTE | "Nintendo [has advised] that it's better to do one iteration on each machine. We were a bit too early, we should have waited for [the next console]." – At the risk of reading too much into this, Ubisoft CEO Yves Guillemot suggests that Nintendo's next console is coming soon enough that it would have made more sense to push last October's underperforming Mario + Rabbids: Sparks of Hope onto the Switch successor instead.

STAT | 0 – The number of previous times in my life I recall gaming hardware getting more expensive as the generation went on. This week's news that Microsoft is raising Xbox Series X prices in most markets, following Sony's PS5 price hike, Meta's upward adjustment of the Quest 2 price, and the Playdate cranking up the sticker shock.

STAT | 37 million hours – The amount of time The Legend of Zelda: Tears of the Kingdom was watched by users on Twitch in May. That was enough to make it the ninth most watched game of the month. People watched the top game, League of Legends, for a combined 124 million hours.

QUOTE | "I think in a post-modern world where nihilism and so on is even more rampant, this is not the path to go forward. I think we have to dive into convictions and beliefs again more, and this would become then what you call a post-post-modern world where all these ideas become more important again." – Lesser Evil founder Hendrik Lesser explains why he wants game makers to take a stand and say things with their work rather than invoke political subjects but just make everyone involved crooked and amoral.

QUOTE | "One attendee raised the issue of diversity and that his organisation, which developed first-person shooters, 'didn't hire women because women don't play first-person shooters'." – In an editorial piece, Skillful's Gina Jackson demonstrates lingering problems with the way the industry perceives and treats women, and calls upon men to speak up against it more.

QUOTE | "May contain intense violence, heavy realistic blood, moderate crude humor, romantic themes, unplayable gambling content and/or the presence of alcohol." – Roblox's explanation of what will be allowed in games for ages 17 and up on the platform, where 43% of users are under the age of 13.

STAT | 30% - The tax break offered by Australia's new Digital Games Tax Offset for studios in the country working on projects with a budget greater than AUD$500,000 ($334,000).

QUOTE | "The global gaming audience is more receptive, or at least accustomed to, branded content and media collaborations within games." – Newzoo's latest survey found gamers love brands more than the average person. Kind of worrying to think there's any activity that produces a Pavlovian response among the participants that is simply pro-brand.

QUOTE | "Our main industry is in decline so we need to plug that gap, but also for the future to bring in these new customers, and get them off of online retailers." – GAME managing director Nick Arran acknowledges that the physical game business is on the ropes as he tells us the retailer's strategy for keeping the doors open.