The race to develop VR controls

Patrick O'Luanaigh is an evangelist, but not the kind you may expect. He believes in virtual reality and has staked the success of his game development studio, nDreams, on its success. He describes some of his experiences with the technology as bordering on spiritual.

"I've had a go on the best tech around, and I felt bubbles on my hand," he says. "I walked through a waterfall. It was amazing. It brought me the future." O'Luanaigh's a dreamer, his eyes set on the possibilities, though he recognizes the world isn't fully there yet.

Like, for instance, the controllers on the market. Many upcoming VR games will run on traditional game controllers, much like any other game on a PlayStation or PC. But long term, that may not be the case.

"[We're] not even close to the potential this stuff has," says O'Luanaigh. "The dual analog controller has been around for 20 years, and it's just not equipped to truly handle that next big leap."

Tying movement, aim and the player’s perspective to a single direction is so ubiquitous it’s practically a given for 3-D games. When you play, Halo, for example, it's common to walk in the same direction that you're looking, and you'll always shoot at whatever is at the center of your screen.

VR flips that. For it to work properly, the headset must be able to detect and respond to the position of the user’s head — effectively transforming the user’s eyes, face and neck into controllers all their own. From there, players can shoot or interact with one object while looking in a totally different direction.

"That's actually our single biggest challenge right now," says O'Luanaigh.

Oculus Rift

Oculus' role

For the past 20 or more years, various companies have experimented with virtual reality headsets. But recently, Oculus VR has become arguably the most prominent company attempting to bring VR mainstream. And to date, the company has shown multiple versions of its headset, but nothing in regards to a new control scheme.

Which makes some developers nervous.

"The problem with the Oculus," O'Luanaigh says, "is that they don't have standard controls established yet. Some developers are using standard Xbox controllers, which is fine. But others are trying to use the mouse and keyboard, which is the worst device known to man for VR. Trying to mash the 'R' key when you can't see at all just isn't an elegant solution. ... Oculus needs a system that allows you to control something with your hands."

When Oculus was the only company producing advanced head-mounted displays for general use, everyone knew what they had to work with. The emergence of other companies producing a dozen or more other kinds of sets, as well as peripherals like the Leap Motion hand sensor or the Virtuix Omni directional treadmill have added a level of confusion and uncertainty that wasn't there before. On the other side are companies like HTC and Sony, who are working not only with integrated controllers, but entire systems that have standardized inputs.

It's far too early to tell which of these technologies will ultimately emerge as dominant. O'Luanaigh likens it to the early stages of flight, where early pioneers tried and tested all manner of wacky shapes before people collectively settled on a few basic airplane and blimp designs.


Setting the standard

Razer, already an established gaming peripheral company, hopes that it can provide a stable platform that allows creative minds to test and experiment free of the fear of obsolescence. Chris Mitchell, a product manager at Razer says that the company is aiming for "universal compatibility."

Contrary to what the name might imply, OSVR is not an operating system per se. Instead it's closer to something like DirectX — software that streamlines communication between video games and graphics cards, making development a bit easier. OSVR handles communication between whatever peripheral a customer might have and the code of a game. In the same way that you can simply download the latest version of DirectX and be reasonably sure that your graphics card, regardless of manufacturer, will be able to play that game, OSVR translates inputs from a multitude of systems, adapting them to suit the developer's purpose.

Chris Mitchell, a product manager for Razer working on OSVR, believes that virtual worlds will live and die by the methods we use to interact with them. "The problem is that as a developer you have no idea what you can count on your users having," he says. "Motion tracking is the first part of it. You need some kind of head-tracking and body-tracking. The problem is that there's a million different ways to do that. You can use cameras, accelerometers, gyroscopes, magnetic field sensors — all manner of things."

"You can try to develop for every piece of hardware and likely bankrupt yourself in the process, or you can partner with someone who develops one basic platform that you can work with."

Razer's first attempt to create its own motion controller, the Hydra, uses magnetic sensors. Mitchell is the first to admit that it's not perfect, though. The three-piece controller connects two handheld remotes to a base station with short tethers. Those wanting to use the device to interface with the virtual face a challenge then, as you quickly run into its limitations when trying to move about. That, Mitchell says is another big problem. Everyone's in such a rush to make peripherals and devices that can let people explore virtual reality that they forget all of this tech is in its infancy.

In the interim though, game makers want some measure of certainty that they won't be writing code for hardware that will fall into disuse a few months down the line. That's the niche Razer's OSVR is looking to fill.

"So you can approach this problem from one of two ways. You can try to develop for every piece of hardware and likely bankrupt yourself in the process, or you can partner with someone who develops one basic platform that you can work with. If you do the latter, you divorce yourself from the hardware choices of the consumer." That's the key, Mitchell believes.

No one knows how quickly peripherals will begin dropping in price, or whether the consumer market will ever be completely ready for something like an omnidirectional treadmill. Standardization, at least partially, will keep everyone on the same page. "PCs didn't become what they are without companies coming to an understanding about how we use them, and what basic parts to include. This is no different."

leap motion controller

Giving players their hands

As part of Razer's Hacker Development kit — a cheap and simple entry point for VR development — it selected a hand tracking device from Leap Motion as an embedded motion controller. The company sees this as an opportunity to give users proper use of their hands in a virtual space.

"Your hands and fingers are the original human interface," says David Holz, co-founder of Leap Motion. "No matter what you do these days, you're probably using your hands. We see hands as the fundamental way to interact with things, and we want to give players access to that in VR."

Leap Motion's controller might be familiar to anyone who has used Microsoft's Kinect. It operates on some of the same principles, and includes several infrared cameras to track the position of hands and fingers in three dimensions. The Leap Motion sensor does this with an incredible degree of accuracy, which Holz says is important because it helps the brain feel that a VR display is merely an extension of the player's body.

"Now that technology is faster and more powerful, we're getting to a point where we can interface with machines in a more personal, human way. The limits have always been on the inputs. You can easily mold a piece of clay in real life, but when you try to do that on the computer, you introduce a separation."

"If we gave you the right scene and then stabbed your hand [in VR] you'll feel really unsettled in a way that nothing else quite matches."

With most human-machine interfaces, like a keyboard, you're always aware that you're using them. You are cognizant of the fact that your experiences are mediated not only by the computer's ability to interpret your actions and intent, but also by the physical means by which you're connecting to it. The goal for VR is to make players forget. Holz says that there's an uncanny valley of interaction. Where audiences begin to feel dysphoric when every link in that connection isn't operating well. When that chain does work though, it can actually bring people so close to the experience that the brain will react to the scenario in unexpected ways.

"If you have a cube that's just floating in space and you touch — and everything's just right with the cube spinning and moving away — then you can feel that sensation. It's a bit like phantom limb syndrome. Similarly, if we gave you the right scene and then stabbed your hand [in VR] you'll feel really unsettled in a way that nothing else quite matches."

Holz believes that this is the precipice of what everyone's been searching for. The technology is there now to create an intimacy between humans and machines. While the absolute ideal of Star Trek's Holodeck is still some ways off, we're not only closer than we've ever been before, but we're closer than we should be.

"The brain is marvelous," Holz says," and it can compensate for a lot if we can just get it most of the way there. There's experiments to back this up, too. We've taken a person's hand and put it under a desk, and then then put a fake hand on top of the desk and then do stuff to the [fake] hand, they feel a bit of that. There are limits, of course. You can't pretend to slap someone and have them feel that, or have someone push against a brick wall and have it suddenly fall away and make that feel realistic. But if you're turning a door knob, then it works. Beyond that, there are some folks working on focused ultrasound so that we can feel textures and have true haptic feedback at a distance. That's really cool, but that's still years away. For now we have to use shortcuts and keep interaction simple to avoid breaking that illusion."

PS Move

The god of dreams

Morpheus is Sony's attempt at making VR work. With it, the company is in a unique position for the coming wars. Morpheus alone ties directly into a popular, well-supported, console system — the PlayStation 4. And it's is capable of using either the standard Dual Shock PS4 controller or the PlayStation Move, a motion controller initially sold as a competitor for the Wii.

The juxtaposition of the cutting edge tech of a VR headset like the Morpheus with the now five-year-old Move might seem odd, but Sony's Richard Marks, believes it's a great fit. "The Move was originally designed to work with the PS3 camera, the PlayStation Eye. And it works better with the PS4 camera ... the new camera's higher resolution and stereo, so you get 3-D automatically. It’s not like there’s a thing with the Move that's like, 'Oh this needs to be fixed.' It’s more, ‘If we do change this, we want it to change in a pretty dramatic way,’ because right now it is serving the purpose of what we want it to do."

Physically, the Move controller is quite a bit different than devices like the Leap Motion. Because it's something you hold, and you use triggers and buttons like you would on a standard controller, it exists as hybrid of input types in VR. The tactile feedback with the controller body itself more closely links input with output.

"With Move we’ve explored a lot of spatial use," Marks says. "So the very simplest thing is I can reach out and grab stuff and pick it up. And that works really good to have an analog trigger, for example. ...

"Just hooking up running to analog sticks isn’t the best feeling in VR."

"[We also experimented] a long time ago with Move; we made a demo where you could grab this branch. You were a chameleon and you saw chameleon hands, and you could grab a branch and pull yourself along the branch. You’re either pulling yourself through the world or you’re pulling the world towards you, depending on the way you look at it, but you could have a lot of motion and not feel sick because it’s your own hand completely doing it."

Sony's also taken a second look at how to incorporate its standard PS4 controller, the Dual Shock, into virtual worlds, while avoiding some of the same problems that have plagued so many others. "The thing that everyone has been discovering is just hooking up running to analog sticks isn’t the best feeling in VR. You can do it, but you have to be more careful than you do on a 2-D TV set."

That's not to say that a regular control won't work, just that it comes with a few more challenges. Chief among them is that there's simply too much mental distance between the player and their world most of the time. Without the direct connection of real-world and virtual movement, the brain can become confused, and then it's the same as any other kind of motion sickness.

Sony's polemic is to bring the controller into the virtual space with the player. "Seeing the Dual Shock down there in your hands, and then seeing a cockpit or a car frame or something around you really grounds you," says Marks.

The key is to make the controller somewhat visible in a way that doesn't break the feeling of presence. That's done by having it "transform" or convert into different kinds of objects. "Now you can go through the world and feel pretty good. And the Dual Shock connects kind of the virtual to the real in a way."

Dual Shock 4

All in the wrists

At time of publication, the only clear understanding developers have is that motion controls will play an essential role in the future of VR. Whether it’s the Oculus Rift, HTC's Vive, the Hacker Dev Kit from Razer, or the Morpheus system for PlayStation 4, all use some variety of motion control.

There are still quite a few unknowns with questions like "How will developers best utilize the new tech?" and "How long it will take consumers to adopt VR?" chief among them.

But there's a powerful wave of optimism driving this industry forward. O'Luanaigh says that VR developers have created a niche that "feels like the good old days, where we're all concerned with how to make the coolest experiences instead of 'How can we chop this up into DLC and sell millions of copies?’"

Part of that is because, he says, that the competition isn't between VR developers, but "VR and not-VR." Every time anyone succeeds at all, the VR companies all share in that success. This has created an environment of positivity where great minds feel free to experiment, iterate, and test new methods of connecting man and machine, he says — all in the pursuit of the transcendent experiences people like O'Luaniagh, Mitchell, and Holz know are out there.

"It's the most exciting time," O'Luanaigh says, "We're still facing a lot of challenges, but we're genuinely on the edge of the sci-fi fantasy of VR, the kinds of things you see in Snow Crash or Ready Player One. That's just around the corner." Babykayak

  • Page 13 of 14