The making of PlayStation VR

When Richard Marks was in 10th grade, his father opened a video game store that was ahead of its time. It was the early ’80s; used game sales weren’t common, and the store — Video Exchange — ran on the gimmick that customers could trade in their games.

The store didn’t last long. But for the young Marks, who held the title of chief demo officer, working there was like living in an arcade. Surrounded by open boxes, he played everything and recommended the best games to customers.

Atari 2600 games. Intellivision games. Anything on a console.

"The most exciting thing [was] when we’d get new hardware," he says, citing the ColecoVision Super Action Controller as an example. "We’d get so excited. And a lot of times the hardware didn’t deliver up to the expectation. ... But it’s still exciting anyway. Your memory somehow forgets that the previous piece of hardware wasn’t quite what you wanted."

Today, Marks is one of the public faces behind PlayStation VR, Sony’s entry into the increasingly crowded virtual reality market. He’s one of hundreds of people who have worked on the headset over the past five years, shaping it from a hacked-together amalgamation of off-the-shelf parts into a futuristic-looking headset appearing in fashion magazines.

After a long development road, the headset is now scheduled for release later this year, and Sony recently invited Polygon to its PlayStation headquarters in both the U.S. and Japan to meet with Marks and other key figures, to learn more about how Sony’s VR project came to be.

With any luck, Marks says he hopes it lives up the the expectations players have built up in their heads.

Magic Lab

In a company of more than 7,000 people, no one person brings a VR headset to market.

Across multiple departments and offices around the world, including help from external partners, Sony’s PlayStation VR team spans one of the widest swaths in the game industry. Someone who works on the 3D audio may never meet someone on the display team, or someone in legal, marketing, business development, hardware design or any of the other groups contributing to the headset. PlayStation executive vice president Masayasu Ito estimates that of PlayStation’s more than 7,000 employees worldwide, in one form or another, 20 percent have contributed to PlayStation VR.

But the spark that lit the fuse began with Marks and a controller his team released in 2010.

Richard Marks
Richard Marks

Since starting at Sony in 1999, Marks has worked on a variety of experimental ways to interact with PlayStation games. Some have taken off, such as the EyeToy camera that let PlayStation 2 players see themselves on the screen and play with simulated objects floating around them, like a weatherman on the evening news. Others never made it to stores, like a way to connect Sony’s AIBO robot dog to PS2 and train it with a controller, or a high-end camera peripheral similar to what Microsoft released with Kinect.

In 2016, he’s the director of a group called "Magic Lab" — a formalized name for what he and two others have been doing for many years, with more freedom.

They work on "pie in the sky" ideas and spend their days looking at what technology research is happening in the world, thinking about how it might apply to PlayStation. Magic Lab is researching eye tracking, for instance, to see if that’s a practical way to control games.

Sometimes the group develops projects with no commercial intent. For a PlayStation 4 launch event, Marks and his team made a version of the classic game Breakout that attendees could play on the ground with their feet. For a sales conference, they made a game that 300 people could play simultaneously on a 100-foot screen. The team’s cubicles resemble a science lab, with a miniature drone, 3D cameras, Nerf toys, an Emmy and a telescope scattered around. On the day of Polygon’s visit, Marks shows a television screen that appears white to the naked eye, but when he holds a magnifying glass up to it, reveals a PS4 menu running on the screen.

"We wanted to call it Magic Lab so you understand they might not all come out as products," he says. "They’re supposed to be kind of experimental."

Marks says that part of his team’s job is to serve as the glue between different divisions of Sony, to keep them all up to speed on what other groups are attempting and to help out when certain groups start to push something forward.

Richard Marks PSX 2015Steven Osman demonstrating a Magic Lab tech demo at PSX 2015

"If you think of a chemistry atom, there’s the nucleus and those are the product groups — they have resources and they have a lot of mass," says Marks. "Then you have the electrons, which have no mass, have no resources, but they bounce around all the time from atom to atom and they share information. And that’s what we are as Magic Lab. We try to communicate between all the groups, and sometimes we can maybe influence them a little bit and we get help from them."

Given Magic Lab’s nebulous structure, its fingerprints appear on many projects — PlayStation VR among them. But while Marks has spent plenty of time researching ways for players to interact with games while wearing the headset, and often makes public appearances to promote it, it’s not his baby in the same way as something like EyeToy, where he came up with the idea and saw it through to release.

His team created the thing that helped make it possible, though.

Prototypes

Prior to forming Magic Lab, Marks and his team took a similar experimental approach to their work. And one of their biggest projects was a motion controller called PlayStation Move. It looked like a wand with a glowing ball on top, and players could swing it around to control what happened on the screen.

Conceptually, it wasn’t a new idea. In 2006, Nintendo had released its Wii Remote, which quickly became a cultural phenomenon as a motion controller that appealed to an audience that didn’t normally play video games. And while Move marked a technological step forward, allowing for more precise 3D control, many saw it as a copycat version of Nintendo’s remote.

Motion controls had become the industry’s biggest trend. Nintendo had the Wii Remote. Microsoft was about to release Kinect. And both companies had gone all-in, spending enormous amounts on marketing in an attempt to reach a new audience.

Sony took a relatively conservative approach with Move, giving it resources but not betting the company on its success.

Jeff Stafford
Jeff Stafford

"If you have a new idea and want to see it turned into something, most people think, ‘Why doesn’t the company just jump suddenly when I say this new idea?’ But there’s a lot of other things going on," says Marks. "There’s a lot of other things you could be making. And you have to really understand how it fits in and how it makes sense."

Following Move’s release in 2010, the controller made money for Sony, but never captured the public’s attention like Wii or Kinect, and didn’t drum up enough software support to sustain long-term sales.

Behind the scenes, though, engineers at Sony realized that the technology inside the Move controller could serve a secondary purpose, giving it a second chance at success.

"The story is complicated," says Marks. "But we basically made Move, and then because Move existed, people could just take Move and mount it to their head and have a tracking system for free."

Jeff Stafford, an engineer in Sony Computer Entertainment America’s R&D division, was one of the first to do so. Marks calls him "the architect" behind PlayStation VR, but Stafford defers credit.

"I’m going to disappoint you," Stafford says. "There’s no one person really you could say, ‘Hey, he’s the architect.’ [Marks] keeps telling me I’m the architect, and I’m like, ‘No, no. I’m not.’"

He describes PlayStation VR as a "grassroots project across multiple divisions."

Stafford currently holds an unusual position within the R&D division. Officially, he’s a senior staff software engineer, though he’s also a co-founder of PlayStation VR and says his job these days is to step back and take a big-picture view of everything related to the headset.

"You could probably say I’m the most vested person in [PlayStation VR]," he says, "because I’ve been doing it probably for the longest, and I’ve been involved in every single aspect of it."

Stafford’s VR story starts in 2010. After spending years on the disease research project Folding@home and the PSX console, which combined a PS2 with a digital video recorder in the same box and only shipped in Japan, he was writing PlayStation internet streaming software for companies like Hulu and Major League Baseball. He had been working on that for a couple of years and was getting tired of it, and Sony had a "10 percent project" where staff could spend 10 percent of their time on whatever they wanted, as long as it had a loose connection to PlayStation.

"I’m going to disappoint you. There’s no one person really you could say, ‘Hey, he’s the architect.’"

He decided to spend that time researching augmented reality technology, seeing what existed, visiting external companies to talk shop and making a website collecting what he learned. Then in 2011, Crusoe Mao, who heads up Sony’s peripherals department, noticed Stafford’s research and suggested Stafford switch his focus to virtual reality.

As it turned out, Mao had been in touch with a small team of Sony’s working out of the U.K. now known as the Immersive Technology Group, which had been looking into a PlayStation virtual reality headset. The group had previously implemented stereoscopic 3D on PlayStation 3, and had experience in the head-mounted display field, with a team member who had worked on headsets for British Aerospace.

Following an introduction from Mao, Stafford started collaborating with the U.K. team, sharing research notes and looking into what might be possible with different lenses and optics. "They were the experts," he says. "I was coming into it just passionate and just wanted to do something fresh."

As time went on, Stafford began to work with a mechanical engineer in the U.S. office named Glen Black. Stafford and the team in the U.K. gave Black various specifications and asked him to mock up units to see what they would look like, building early prototypes. Stafford describes them as "viewer boxes," meaning the developers hadn’t yet incorporated tracking for the headset, which at one point the team codenamed "Project Morpheus."

PlayStation VR prototypesPlayStation VR prototypes

Then, Stafford says, inspiration struck. In January 2012, he attended the Consumer Electronics Show in Las Vegas, looking through the masses of booths for anything related to a virtual or augmented reality headset. Off in the distance he saw what he thought was a head-mounted display. When he got closer, he realized it was actually a head massager. The disappointment soon passed, and he realized that he could use the massager as essentially a sturdy hat to build a VR headset prototype on.

"It was bulky enough that you could bolt things on and counterbalance it," he says. "So I ordered that from Amazon and we built our first official Morpheus 1.0 prototype."

Stafford keeps a pile of PlayStation VR prototypes in a drawer next to his desk. There’s a version 1.1, a version 1.5, a version 2. Each one tested out things like different optics, a wider field of view or a clearer screen.

"When we announced, the version that we showed to the public, that’s what we officially said is our first version, right? The one that was at GDC. But, you know, of course when you build a product — any product — there’s so many internal versions. And at some point you have to stick a label and say, ‘This is [version X].’"

Interactive 3D model

In an effort to show every angle of PlayStation VR, Polygon asked 3D artist Jan Neves to design a simple version of the headset for our readers. Check it out above to rotate the hardware, zoom in on specific parts and read notes on how Sony’s engineers designed the headset.

Stafford says that one of the benefits of working at Sony on a project like PlayStation VR is that some of the technology needed to build it already exists around the company. A group had been working on 3D audio, for instance, so that became available to the VR team. And Marks’ team had the Move controller, which the VR team ended up sticking on the head massager as a starting point to track player head movement.

Given the scope of the PlayStation team at Sony, Stafford wasn’t the first at the company to attempt a virtual reality headset, or even the first to stick a Move controller on one for tracking. Unbeknownst to him, others around the company had also been experimenting with headset ideas, such as a small team in Sony Santa Monica’s office that developed a first-person God of War VR tech demo running on PS3. These teams eventually started sharing notes, feeding into the prototypes Stafford and the U.K. team were putting together.

At a certain point, Stafford and team had progressed to the point where they felt they had something good enough to pitch up the chain, in the hopes of one day turning it into a salable product. They booked a flight to Japan to visit the executives at PlayStation’s head office in Tokyo and met with, among others, PlayStation hardware boss Kazuo Miura, who liked what he saw and volunteered to take the project under his wing.

By mid-2012, Stafford says, his team got a green light to pursue the headset as a product. With Sony Computer Entertainment Worldwide Studios president Shuhei Yoshida and Miura overseeing the project, Sony began putting a larger team together, right around the time an upstart company called Oculus VR started making noise about a headset of its own.

Refinement

When Oculus went public with its plans for a virtual reality headset in 2012, the world took notice, kicking off a wave of mainstream interest in the idea. Soon after, others also announced plans for headsets. Behind the scenes, Sony kept moving forward with its plans, growing its internal team working on PlayStation VR hardware and creating a joint R&D team across its U.S. and Japan offices.

While Stafford and the U.K. team had their hands in many pieces of the puzzle, they weren’t in a position to oversee the larger team. On the hardware side, that role went to Miura — and currently goes to Masayasu Ito, who took the baton from Miura in 2014. Ito joined Sony in 1986, initially working as an engineer in its car stereo department before moving over and overseeing the PS3 and PS4 hardware, taking a heavier role in the business side as time has gone on.

Masayasu Ito
Masayasu Ito

Ito also works closely with hardware director Yasuo Takahashi, who started his career working on mobile phone screens and has been on PlayStation peripherals since 2010. Between Miura, Ito, Takahashi and many others, Sony’s team gradually took shape.

As things got rolling, the team experimented with whatever it could think up. Different screens, different fields of view, different refresh rates, different methods of reducing latency, different visual designs. Some team members suggested sliding in the PlayStation Vita for the screen, similar to what Oculus and Samsung ended up doing with mobile phones in their low-budget Gear VR headset, but that idea didn’t last long because the Vita’s 960x540 resolution was too low for the company's needs.

When Ito took over the hardware in late 2014, he says, he found the prototypes difficult to put on and easy to get lost inside. Under his leadership, staff members continually tweaked the headset’s design and put it through a series of user tests. They removed rubber straps on the inside to make the headset easier to put on, and added a gap so players could see the floor while wearing it, in case they felt overwhelmed or needed to grab something in the real world.

The team spent time iterating on the headset’s comfort, testing "goggle" prototypes that strapped most of the weight to a player’s face before settling on the current "visor" design that places more weight on the player’s forehead, with additional weight in the back to counterbalance the unit. Takahashi says that adding this weight in the back makes the headset feel lighter overall, because the player doesn’t notice a strain in any one area.

Ito says his two top criteria for the headset were to make it affordable and to make it easy to put on, and he’s happy with the results. The team ran into certain limitations, though. Having a cord sticking out of the headset and having to put on an additional pair of headphones (with its own cords) leads to extra wires and loose parts.

Yasuo Takahashi
Yasuo Takahashi

The staff experimented with built-in headphones, but came to realize that with as many different head shapes as there are in the world, it couldn’t design headphones that would properly cover everyone’s ears, so it decided to give players the ability to use their own.

As all this was going on, other parts of the team devised safety and comfort recommendations for PlayStation VR software. Team members went through debates over whether to allow software to ask players to stand up, ending up recommending that games don’t do that but allowing it under limited circumstances.

The team started building a list of best practices for developers, things like not requiring players to turn around quickly, not showing objects flashing close to players’ faces, and not designing experiences where players spin in circles to the point that they get tangled in the cord. Going one step further for safety, team members developed a warning system where if the PlayStation Camera senses a player is moving out of range, it sends them a message in the headset.

While the R&D group refined the hardware design, it also took units on road shows to drum up interest and to get feedback on what game development studios would like to see in a headset. In part because of its proximity to the hardware team in Tokyo, Sony’s Asobi! team, which develops the Playroom series of experimental minigames, ended up being one of the main sources for developer feedback.

The small group works within Sony’s Japan Studio, which is located in a large building in Tokyo’s Shinagawa district, housing the teams that work on games like The Last Guardian and Gravity Rush 2. A couple blocks away, many of Sony’s top executives like Ito and Yoshida work in the corporate Sony City building, with two floors dedicated to PlayStation staff.

Nicolas Doucet
Nicolas Doucet

Asobi! team producer Nicolas Doucet and senior programmer Yutaka Yokokawa say they frequently walk back and forth between the two buildings. When Polygon visits, Yokokawa says he is about to head over to Sony City for a weekly feedback meeting.

The two have been working on PlayStation VR software experiments since 2014. One of the first involved modeling the PS4 DualShock 4 controller inside a game so when a player looks down at their hands in the game, they see a replica of the controller, which worked well for the team and led to other experiments. One of those took the in-game controller replica and used it to pull the player through the sky, but the team discarded it after realizing that the flight movement made some players nauseated.

"For those who can withstand that, it’s really amazing," says Doucet. "But we couldn’t bring it down to a level where it feels right for everyone."

In Doucet and Yokokawa’s position, experimenting with PlayStation VR games close to the hardware team, they put in a variety of suggestions for the headset.

One of their top requests was the ability to split the video signal from the PS4 so a game could appear not only in the headset, but also on a television at the same time. In a certain sense, this would be a marketing tactic to show onlookers games they wouldn’t otherwise see, but for Doucet and Yokokawa, it would also allow for local multiplayer games. They envisioned games where one player would wear the headset and others would sit next to them, competing or collaborating on the TV.

So they pitched it to the hardware team.

"They weren’t really sure, but we really strongly requested for it and it became real," says Yokokawa.

Move the slider to see a Playroom VR prototype scene (left) next to the near-final product (right).

The hardware team implemented this "social screen," as Sony now calls it, by way of a processing box that sits between the headset and the PS4, and splits the video signal so it appears in two places.

And now, Doucet and Yokokawa’s team has multiplayer minigames running in Playroom VR, such as a race (seen above) where players without the headset control small robots who run away from and then attack a monster, controlled by the player in the headset. Or a Ghostbusters-style game where the player in the headset can shoot a beam to capture ghosts, but only the players not wearing the headset can see the ghosts’ locations, so players inside and outside the headset have to collaborate.

Playroom VR is one of many experimental projects Sony's teams have attempted. As the hardware team got closer to the finish line with the headset — eventually announcing it publicly in March 2014 — the software teams started ramping up their efforts.

The games

In Shuhei Yoshida’s day job, he oversees Sony’s internal development studios and often gets to deliver good news to fans. When Sony announced the return of long-in-development fan favorite The Last Guardian at E3 in 2015, he was on stage to do the honors.

But he also works as something of an evangelist for PlayStation VR, doing interviews and talking to developers about what’s possible with the new hardware.

He likes to say that Sony is treating PlayStation VR like a console launch rather than a peripheral launch. As it requires a PlayStation 4 to run, the headset is inherently a peripheral, but its complexity, presumed cost and extensive game lineup make it a larger proposition than most. The game lineup already outnumbers the lifetime number of releases for peripherals like EyeToy and PlayStation Move.

Shuhei Yoshida
Shuhei Yoshida

Internally, Sony is taking what Marks refers to as a "shotgun approach" to software, investing in a broad group of small experiments rather than placing big bets on a few key titles.

"At this point, VR is about creating new experiences, and we don’t need as much content — asset creation," says Yoshida. "That’s where it takes the largest number of resources."

Yoshida says that at the moment, Sony doesn’t have any teams of more than 100 people working on a single VR game, on the scale of something like Uncharted 4: A Thief’s End.

But, he says, "that will naturally happen."

"It’s just the relative progression of any new media," he says. "When the content side starts investing, you usually start small. And with VR, small games can have a very significant impact, and that’s more preferable. I’ve been saying that [to developers]: ‘Don’t start to write big design documents. If you do that while you’re working on something long-term, the whole industry will learn a lot and you’ll discover half of what you set out to do will become obsolete. So focus on finding great experiences, and packaging it to deliver, and keep doing that.’"

Currently, PlayStation VR’s roster matches Marks and Yoshida’s descriptions.

Yoshida says that of Sony’s internal development teams, its London Studio has "probably" made the largest push toward VR, with tech demos The Deep, VR Luge and The London Heist appearing at promotional events. However, Yoshida teases that none of those have "yet" been announced as proper games.

The London HeistSCE London Studio’s tech demo The London Heist

Meanwhile, some of Sony’s teams working on traditional PS4 games have begun to add virtual reality features, like in build-your-own-game tool Dreams and the long-running racing series Gran Turismo.

The majority of games announced for PlayStation VR come from external studios, though, some debuting on Sony’s headset like relationship simulator Summer Lesson and action-adventure game Golem, and others also heading to competing VR headsets.

Regardless of developer, most PlayStation VR games seem to be following Sony’s plan of experimenting small and seeing what works, rather than throwing big money into major projects. Yoshida says PlayStation VR games will sell for similar amounts to regular PS4 games, with cheaper digital titles priced in the $10 range and higher-end boxed games priced in the $60 range.

Sony has yet to reveal the price of the headset itself.

The price

Traditionally, PlayStation’s direct competitors have been Microsoft and to a lesser degree Nintendo, and the trio have held a strong grip on the console game market. Now with PlayStation VR being one of the first high-end virtual reality headsets arriving in 2016, Sony has two new competitors from the broader PC and tech industries.

Oculus’ Rift is the well-funded virtual reality poster child, while HTC’s Vive comes from Steam creator Valve and features some of the highest-end VR experiences to date.

Sony’s approach, relative to these, is to be in roughly the same conversation on the tech front, but to be the cheaper option.

Nicolas Doucet
Nicolas Doucet

Sony has said PlayStation VR "will be priced as a new gaming platform," according to a Bloomberg report from 2015, a line Yoshida repeats for this story. He declines to elaborate until Sony reveals the official price, though Sony executives repeatedly use the word "affordable" to describe the unit, and many speculate it will land in the $300-$400 range.

Meanwhile, Oculus' Rift costs $599 and HTC’s Vive costs $799. Yoshida says that like much of the public he was surprised by Oculus’ price announcement, given the company’s early public statements suggesting the Rift would cost less.

A potentially bigger difference comes with the hardware required to run the headset. PlayStation VR requires a PS4, which currently retails for $349, while the Rift and Vive require PCs that can cost closer to $1,000.

Added up, Sony’s proposition is significantly cheaper, though the company had to make certain tech trade-offs to get there. PlayStation VR features a smaller field of view and lower-resolution screen than the Rift and Vive, for example (though Sony staff tout their unit's 120 Hz refresh rate as an advantage).

"If you just talk about the high-end quality, yes, I would admit that Oculus may have better VR," says Ito. "However, it requires a very expensive and very fast PC. The biggest advantage for Sony is our headset works with PS4. It’s more for everyday use, so it has to be easy to use and it has to be affordable. This is not for the person who uses a high-end PC. It’s for the mass market."

He says this is a direction that has been handed down by Sony Computer Entertainment president and global CEO Andrew House, who has been at Sony for more than 20 years and seen PlayStation’s highs and lows.

"If you just talk about the high-end quality, yes, I would admit that Oculus may have better VR. However, it requires a very expensive and very fast PC."

With PlayStation 3, Sony launched at $599 and struggled to succeed at that price. With PlayStation 4, Sony launched at $399 and has had the fastest-selling console in the company’s history. That difference reflects more than price, but when asked whether Sony’s approach has changed on a philosophical level in that time frame, Ito says he thinks it has.

"We would not say that PS3 was a failure; however, there are many things we kind of regret," he says. "And we took the things we regretted with PS3 and made PS4 on top of that. So therefore our development of PS4 and VR is in some ways a reaction to when we had a hard time with PS3."

With the PS3 nearing its 10th anniversary, Ito says he’s happy with how the PS4 has performed in comparison. In January, Sony revealed that it had sold 35.9 million PS4 consoles by the end of 2015.

The future

As someone whose job consists of looking at tech coming in the future, Marks isn’t sure how PlayStation VR will evolve over the next five years, but he sees Sony in the market for the long haul.

He’s confident virtual reality will stick around, and hopes that games help lead that charge. Given his background in exploring how players interact with games in new ways, it’s perhaps unsurprising that he doesn’t want to see VR become just a new form of movies that take people into their worlds without giving them things to do.

Richard Marks
Richard Marks

"I really hope that we stay equal to the other kind of content that gets created for VR, which is more passive. Not just Sony, but as an industry I hope we don’t just sit back and say, ‘Oh, it’s good enough to just look around.’ Because [I think] that’s undercutting what VR could be."

For now, he sees Sony continuing to research new control options like eye tracking, and ways of making players more aware of their entire body when in VR. At launch, players will be able to control PlayStation VR games with the standard DualShock 4 controller, the Move controller or just by looking with the headset itself. But down the road, Marks says, he wouldn’t be surprised to see other options on that list.

Nor would he be surprised to see another PlayStation VR unit altogether.

"We definitely believe [there will be a follow-up someday]," says Marks. "We’re doing everything and planning and setting it up so that we would know how to make a next one, and make that make sense for our customers and stuff, so yeah, we don’t think of this as a one-and-done by a long shot. And we also don’t think VR is like that."

First, Sony has to get PlayStation VR into stores. As of this story going live, it hasn’t announced a release date, while Oculus’ Rift ships this month and HTC’s Vive ships in April.

When Sony’s unit ships, it will mark the end of a more than five-year road for some like Stafford. In another five years, Stafford says with a laugh, "I’ll probably still be working on it."

"We’re going to try and improve every aspect of it," he says. "Get it lighter, smaller form factor, easier to use. Of course people want wireless — that’s a challenge. There’s so many things. People want to see their hands in VR, or see their body in VR. ... You get all of that [and then] ‘OK, I want to be able to feel things in VR.’ It’s the next thing. ‘OK, can I smell this virtual coffee?’ You know, at some point many years in the future I’m sure there will be some new amazing technology where it gives you a lot of these things. But even now, there’s so many experiences that can be created with the current technology. We haven’t really seen any of it yet. We’ll start to see it soon." Babykayak

Photos: Jonathan Castillo, Kelsey Floyd, Karl Nielsen, Irwin Wong