Sections

VR and AR will converge, says Unity XR lead

Timoni West foresees a future where VR/AR/mixed reality technology outpaces consoles, offers alternative to PCs

The gaming industry may be conflicted on the future of VR and AR, but as director of XR at Unity, it's Timoni West's job to remain optimistic.

And it's a job she seems to find effortless, especially as she's constantly working with the developers who are pushing innovation into the technology.

"Since the advent of what people are calling the third wave of virtual reality, Unity's numbers have been [strong]. I think 60% of all VR content is made with Unity," she says. "The number is much higher for augmented reality, with 90% of all Hololens content, for example. And obviously, we're always in the mobile space. But the cool thing about virtual reality specifically is that when it came time to make these 3D interactive worlds, Unity had been doing that for years, but people just hadn't had the headset on which to do it."

West speaks with pride as she tells me that Unity has been involved with VR from the very start in its support of developers working on every headset on the market. And it plans to keep that level of involvement with AR as the technology blossoms, a plan that led to the announcement that AR Foundation was moving into preview at GDC 2019.

"The reason why AR only came about in the last three years is because we finally made small, portable devices that were fast enough to take in vast quantities of information...and do something about it"

AR Foundation is a Unity engine package that allows users to develop for AR across both ARCore and ARKit, with plans to add any other future platforms. This means that developers making AR games for one platform don't need to completely rework their creations for a second platform - they simply work across both.

West says this kind of ease is key at this particular moment in time, as AR is just now beginning to come into its own thanks to the advent of mobile devices capable of using the technology.

"When it comes to augmented reality, the key difference between it and virtual reality today is how much 'world data' that you need to take in in order for augmented reality to work," she says. "When it comes to virtual reality, you're tricking the human brain into thinking it's somewhere else. But when it comes to augmented reality, you need to have your digital experience work in the real world. The reason why augmented reality only came about in the last three years in a real way is because we finally made small, portable devices that were fast enough to take in those vast quantities of world information from camera feeds or wherever, and be able to process it, get semantic information out of it, and do something about it."

West is pleased that there are effectively two fields for VR headsets. On the high-end side, headsets like the Valve Index allow text to be read - something key for developers

West is pleased that there are effectively two fields for VR headsets. On the high-end side, headsets like the Valve Index allow text to be read - something key for developers

With both VR and AR still in their early stages, much of the development being done in either technology is still experimental and often surprising. Part of West's job is to stay on top of what developers need as they continue to make strides in either one, which she does by listening to Unity's developer partners, and also by keeping an eye on what major platform holders and fellow AR and VR rivals seem to be up to. A recent example she offers is that of 3D markers.

"We already have 2D image markers that have been around for a long time, like QR codes," she says. "The difference between a 2D marker and a 3D marker is that if an object is at an angle, you still need to have it work no matter what angle the device sees it at. ARKit is coming out with 3D object recognition.That's a thing we knew that we needed, and although Apple kept things very close to the chest and we didn't know if they'd have it, it was an obvious use case. So it made it easy for us to say we needed to support it across the board and make sure our developers have access to it in a consistent way.

"There are some things that are further afield that we're not sure how many different platforms will offer. Here's a weird example. You can extrapolate out heartbeat from eye tracking by measuring how the veins in the eye change. That's weird, right? Are we going to support heartbeat? I'm not sure. That's a use case where we know you can do it, but it's a question where, if we hear that the Hololens team is doing it as well, and then if we hear that a studio that is working with a hospital is also supporting this, or that Apple wants to put it in their health app, then we can say it's a compelling use case and make this a key feature we support."

"We got a Nintendo when I was eight, and that thing did not work 90% of the time. People like to make fun of virtual reality, but new manufacturing is always hard"

Keeping an eye on trends and industry conversations also means that West is part of the discussions revolving around challenges in XR development that no one has quite solved. One major one she highlighted was that of object persistence and relocation in AR. Put simply, AR technology still hasn't perfectly solved the problem of 'remembering' digital spaces, such as if you place a virtual object into a room, leave, and then return.

"If I put a giant digital teddy bear in my conference room and I want it to face the door, I need to make sure all the information about the room orientation, my device's orientation, and the orientation of the object in the room are stored, so when I go back in I can pull that information back from the cloud or device or wherever. Object persistence isn't a thing that can exist on its own without a lot of other stuff, which includes information about the room, you, your device, and where the object's last state was.

"There are companies specifically working on this problem, and we can very much assume that a large part of the big partners like Google, Apple, and Facebook are working on this problem independently. A big problem then is how we get all that information to relocalize across all platforms, but that's a discussion that's happening both in the community, internally at companies, and with all our partners."

Problems aside, West's vision of the future isn't just one where AR and VR simply function slightly better than they do now. She's untroubled by cynicism about either market, comparing VR in particular to the boom of consoles.

"I am 38 years old, and we got a Nintendo when I was eight, and that thing did not work 90% of the time. Sometimes it just wouldn't start! People like to make fun of virtual reality headsets today, but come on! New manufacturing is always hard.

"The first home computer console came out in 1968. It had twelve games. That was almost 20 years before Nintendo hit the consumer market of America. And just before Nintendo, there was a big dip in the console market. Computers were taking over. Then Nintendo comes out and they have this fabulous D-pad and some great games and they just totally killed it. But that took almost 20 years. That's a really significant amount of time. I think we're going to catch up a lot faster this time around."

On the consumer grade side, West says headsets like the Oculus Quest allow more people to experience VR, in turn offering more incentive for developers working in it

On the consumer grade side, West says headsets like the Oculus Quest allow more people to experience VR, in turn offering more incentive for developers working in it

In fact, West believes that XR development is about to start moving even faster. VR took its time to get there, she says, but there are now consumer-priced headsets on the market. AR isn't quite there yet, but because VR already has a lot of the mechanisms needed for AR to work, West believes a combination of the two technologies (at consumer-level pricing, too) is on the horizon.

"Augmented reality has all the challenges of virtual reality, and it has to have a bunch of sensors on it so it can take in the real world information," she says. "That's why it's lagging a bit behind. The cool thing though is that even the virtual reality headsets are starting to get sensors of their own, forward-facing cameras so they can do live feeds, so I think we'll start to see a convergence, possibly not in 2019, more in 2020, where these devices truly become more XR-ish."

"People want computers to be different than they are. That's why we're making virtual and augmented reality headsets"

But more than that, West thinks the true uses for XR are still to come. Rather than viewing them as just gaming devices or just industry tools, West's work with a wide variety of developers for a broad spectrum of projects has given her a vision of vast potential for the technology. The future of XR, she says, isn't as a headset you buy for one specific purpose. Rather, it's more akin to how buying a PC is now, with XR technology one day replacing typical screens by virtue of one day being far, far better at what they already do.

"You might spend a lot of time playing games if you're at a certain age, but most of what we do on computers now is really boring," West says. "We type emails to each other, we tweet to each other, Facebook. And right now text is still pretty shitty on lower-end VR headsets. The resolution just isn't that great. Until we hit that point, we might see that people just don't want to spend as much time in VR.

"I'm not that worried about it though, because in the end, people want computers to be different than they are. That's why we're making virtual and augmented reality headsets. We know what we have right now isn't good enough. We don't even like it that much. It's not ergonomic. It's not that accessible. People use computers every day, and they don't get to stretch their imagination, they don't get to express what they want, they don't get to use them to the limits of their intellectual or artistic curiosity. They're constraining little boxes.

"Even if it doesn't take off this year, we're always going to be trying to push this medium forward to that type of computer where anyone can pick it up and do what they want to with it. I truly believe that's the future we're heading towards."

Related stories

Unity to raise subscription prices

Beginning in January, Pro subs will go up 20% to $150 a month, while Plus plans rise 14% to $40 a month

By Brendan Sinclair

Unity acquires ChilliConnect

Live game management platform to join deltaDNA to improve Unity's live ops offerings

By Rebekah Valentine

Latest comments (7)

Dariusz G. Jagielski Game Developer 4 months ago
Saying AR and VR must converge is like saying helicopters and cars must converge. Those technologies simply fit different purposes. Similar, but very different. AR is about putting virtual stuff in the real world while VR is all about putting you in a completely made up place with no input of what's around you. Just like in theory cars and helicopters are similar in the way that they transport you from A to be and usually are 2-4 people, with bigger variants available. On drives on the roads though and the other is flying. You can't converge that.

Not to mention that VR is still too early to market. Yes, it can provide some enjoyment but the headsets and computers capable of running it are still too expensive not to mention that it's literally a screen planted to your face, not a Matrix-like experience everyone would think of when hearing "VR".
0Sign inorRegisterto rate and reply
Joe Steffard VR Developer, Self-Emplyed4 months ago
I'd say you are quite mistaken here. Mixing these together enables you to shift between wanting data at different levels of immersion. It allows what would normally be AR headsets to be more immersive and allows what would be VR headsets to be more aware of your surroundings. You say VR is about having no input around you, but why does it have to only be that? Why can't the people who are wary of buying in because they don't like being disconnected be offered the ability to experience VR that's a little more grounded in reality?

I think the real potential of both VR and AR is that they are ways of enabling us to control reality and life as an extension of your body.

They do fit different purposes, but they also fit similar purposes too as there's plenty of overlap to be had.

A car and a helicopter doesn't make sense to combine because it's a crazy engineering feat and would interfere greatly with occupied airspace. AR and VR don't have issues like this. Headsets can already switch between the two using cameras and perhaps eventually by switching the opacity of glasses.

I think everyone will be happy with the concept of VR after they try it even if it isn't the Matrix. It's just that a lot of people haven't tried it yet. I've demoed it to a lot of people and it converts people into the idea as fast as snapping my fingers. I'm sure if it eventually gets to something as shown in the Ready Player One movie that came out recently, that might be close enough for those looking for some crazy Matrix experience, but I reject the idea that it ever needs to be the Matrix.

Edited 2 times. Last edit by Joe Steffard on 27th May 2019 9:26pm

0Sign inorRegisterto rate and reply
benn Employee, buff up4 months ago
nope.
0Sign inorRegisterto rate and reply
Show all comments (7)
Eyal Teler Programmer 4 months ago
@Dariusz G. Jagielski, I'd say that a lot of Oculus Quest users would beg to differ. I'm still not sure if it's a Wii moment or an iPhone moment (i.e., whether it'd sell a ton because it's cool but will end up gathering dust, or start a real revolution in the way people experience gaming), but weak and heavy and somewhat buggy as it is, it has that wow factor of 'VR has finally arrived'.

By the way, combined car/helicopters are under development, in the form of drone cars.

Anyway, regarding the the article's prediction of XR replacing PC's, I don't think that'd happen any time soon. It'd be a long time before XR is anywhere close enough in terms of ergonomics. That would likely require a mind-computer interface for input (which Facebook is already working on). What I see happening before that is AR replacing mobile. Having info overlayed on the world and movies play on a large screen in front of your eyes will be a huge step up from those tiny screens.
0Sign inorRegisterto rate and reply
Dariusz G. Jagielski Game Developer 4 months ago
@Eyal Teler: "Having info overlayed on the world and movies play on a large screen in front of your eyes will be a huge step up from those tiny screens."
Yeah, right under oncoming traffic. We already have problems with people being basically glued to their phones. We don't have to give them another way of killing themselves without actually meaning to.

"By the way, combined car/helicopters are under development, in the form of drone cars."
And they make about as much sense as XR.

"Why can't the people who are wary of buying in because they don't like being disconnected be offered the ability to experience VR that's a little more grounded in reality?"
You mean, like AR? Again, AR: Some virtual stuff (say, a pokemon) in a real world. VR: Completely virtual world you get immersed in fully.
0Sign inorRegisterto rate and reply
Eyal Teler Programmer 4 months ago
@Dariusz G. Jagielski, "We already have problems with people being basically glued to their phones."

I'm not sure why you think AR will be any worse. As you say, it's an existing problem. Also, I'm not sure if that's supposed to an argument against AR being successful. Certainly phones are extremely successful, and perhaps being a good distraction is part of the reason.
1Sign inorRegisterto rate and reply
Joe Steffard VR Developer, Self-Emplyed4 months ago
@Dariusz G. Jagielski: I mean AVR. Augmented Virtual Reality. It's where you are still in VR but you have the real world reconstructed into the headset. There's a lot more than just a simple AR or VR definition these days. It gets complicated.
0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.