Nimble Sense: Hands on with virtual reality

The hand-scanning tech that lets you touch the VR world

Corner a virtual reality sceptic and there's one thing they'll point to over and over again: how do I interact with that reality? Answers range from simple controllers to complicated treadmills that will look really weird in your lounge, but one San Francisco company has a better answer, and you already own them. Your hands.

Think Kinect, but better. Think super sensitive gloves, but without having to wear the actual gloves. The Nimble Sense is a surprisingly small camera that clips to the Oculus Rift and scans your hands, allowing you to manipulate objects and make gestures in the virtual world.

"The Oculus Rift brought people's eyes into VR, that's great, but we also want to bring the rest of their bodies into VR, starting with the hands," explains Robert Wang, one of the creators of Nimble Sense.

"We want to be part of this journey that people are trying to figure out; how is this all going to work out?"

Instead of seeing colour the camera sees depth, taking a scan of the whole field of view every 20 milliseconds. That data is then sent to the Oculus Rift, for each eye. In practice it's quick, with no noticeable latency issues for the untrained eye. I'm able to push buttons in a floating interface to start and stop films and, in another demo, shoot lightning from my palms. Wang shows me more, a bomb defusing demo that could have been from any first-person game and a simple demo where he manipulates a cup and die.

It's currently on Kickstarter with 22 days to go but has already beaten its goal, raising $104,653, some distance from the original $62,500 request. Scan the backers list and you'll even spot Oculus VR CEO Brendan Iribe on there.

"Kickstarter is about raising money and it's also about tapping into the community. So every developer for the Oculus Rift probably backed Oculus Rift for their Kickstarter; they already have Kickstarter accounts. We can actually tap into those people and say if you're interested in building new interactions, if you're interested in trying out some crazy new hardware, another DK1 in some ways for tracking hands, then get in touch with us."

"This is not the ultimate solution yet, it's a part of it"

Wang and the team wanted to use Kickstarter to gauge feedback and build a community, to tap into the collaborative way of working that seems so unique to the current virtual reality development space.

"We didn't believe in VR for a long time, we had to sort of be dragged into VR," he admits.

"It's only after we started going to some of the meetups, like the Silicon Valley Virtual Reality Meetup, that we were really convinced. You could feel the level of excitement, and how quickly everything was moving. We've been working on hands for a long time and it just seemed like a good fit."

He said the biggest challenge for the project so far was that the premise is a very complicated computer vision problem; it's very technical. The scan not only needs to track your hands but understand what each of your fingers is doing. Hands can move quickly, hands can flip around, fingers can occlude one another. The team are making headway, but they want to deliver something "rock solid" in the next six or seven months.

"This is not a consumer product right now and it's probably a few iterations away from that," says Wang of the Nimble Sense's progress.

"It's only through developers and trying to figure out what the right SDK is for them - what the specs for the camera will need to be to support the interactions they want - that we'll get there. This is not the ultimate solution yet, it's a part of it. The other parts we're still trying to figure out."

More stories

Qualcomm unveils handheld dev kit powered by first dedicated gaming chip

The Android-based platform features 5G connectivity and can run games at 144 frames per second

By Marie Dealessandri

Polymega adds N64 emulation module

Maker of multi-system retro console also planning digital storefront and original content exclusive to the system

By Brendan Sinclair

Latest comments (4)

Julian Williams Founder, WIZDISH Ltd.7 years ago
I love it and hope they do well. It looks like this is the same Robert Wang who did such impressive work in 3GearSystems.
That should mean he isn't just another Kickstarter charlatan.
1Sign inorRegisterto rate and reply
Greg Wilcox Creator, Destroy All Fanboys! 7 years ago
It's the same Robert Wang, Julian. I tried the Nimble Sense out when he was here in NYC and as a VR virgin and still slightly skeptical, it sold me because finally, someone got it in terms of another wall that needed to come down. It's a very awesome thing to see in action (warts and all), but it's also going to spur a lot of developers to really get creative with interaction. I knew it was going to be funded within a week of testing it out. I think it was three days later that it broke its target with time to spare.

Now, someone needs to figure out force feedback in VR without a controller, as I'm gathering the only way to "replicate" it is via an aural experience that has you "feel" impact through use of sound effects.
0Sign inorRegisterto rate and reply
Henry Durrant Programmer, SUMO Digital7 years ago
This would be great even if it just gave you hand-presense feedback ( without the skeletal tracking ).
The only awkward thing I felt when using a DK2 was not being able to see my own hands.

The only issue i see with using a head-mounted version is that you will only be able to interact with things you are directly looking at, with flight-sims for example, your hands are often not in your field of view as you look around.

However, it seems like a great peripheral for certain VR applications and it being a clip-on peripheral means less clutter and not having to stay inside a table-top sensor's scope.
0Sign inorRegisterto rate and reply
Show all comments (4)
Julian Williams Founder, WIZDISH Ltd.7 years ago
I have a feeling that VR interfaces may resemble sport in that each game type/experience will require its own hardware.
Abstract interfaces such as gamepads detract from the sense of presence. However, literal translations are equally undesirable because we don't want to replicate the tedium, fatigue and costs of real life. Only sufficient stimulus to trick yourself into believing you've done something is needed. Hence terms like 'sufficient motion'.
I imagine the commercial Rift or Morpheus will include this feature so this device may be short lived. There is a strong desire to see your hands but you may then point at objects and twist you hands to manipulate them in a semi-abstract fashion. The best way to get tactile feedback will likely be to hold a prop with the same shape and feel.
0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.