Before going home tonight, Eric Schuh will watch Halo 4's ending 16 times.
It will be under less-than romantic circumstances, on blurry six-inch screens with no sound. The code won't be complete. He'll gloss over cutscenes. People will walk in front of him and talk in the background, surgically criticizing the smallest details.
He'll watch players guide series star Master Chief in his fight against new enemies the Prometheans, never once touching a controller.
Schuh doesn't mind. He's done this before.
As the Halo lead in Microsoft's Studios User Research and Central Analytics department, he's spent the past two years collecting data on the much-anticipated sci-fi shooter. He and other team members recruit players, sit them in front of Xboxes, then watch, record and analyze everything that happens. They learn what confuses people, and turn their data into advice to the development team on what to babyproof and what to amputate.
Then they speak in numbers. Out of a database of 60,000 volunteers, Schuh says his team can fit 60 in the three rooms in front of him, 18 are playing Halo today and each will answer 884 survey questions before leaving. He's wearing a shirt with "10 years of consolidated research" on the back, in the form of a word cloud showing survey results, and points out $20,000 worth of art on the walls, showing abstract representations of data.
"We like to keep this as scientific as possible," he says. What some companies call "playtests" he calls "studies," what some call "game rooms" he calls "labs," and what some call "players" he calls "participants."
To date, Schuh has run 40 tests on Halo 4's highly secretive single-player campaign - totaling, as he tells it, 1,252 players and $2,750 in pizza.
Today, July 15th, is the last one.
Ready at Midnight
Schuh's road to this point began in 1992. Enrolled in the University of Washington, studying to be a social psychologist, he "almost but not quite" got his PHD when a friend turned him onto a job opening at Microsoft.
Since 2002, he's been in the Xbox group, touching most games that have come through. Collecting and analyzing data, managing risk and limiting player confusion and frustration.
Schuh's done well for himself. After working on Halo: ODST, Schuh graduated into a management role to oversee many of the projects in the User Research department. But when Halo 4 came around — then known under the codename "Midnight" — he stepped down from that role to focus on it.
ASKED IF IT'S THE MOST EXPENSIVE GAME MICROSOFT HAS MADE, HE SAYS, "ABSOLUTELY. NOTHING'S EVEN CLOSE."
That might seem counter intuitive to an outsider, but as he explains it, for someone who works in data collection and user research analysis, it's tough to find a better gig than Halo 4.
"The allure of not eating Top Ramen was pretty high," he says.
On the surface, it's Microsoft's most popular game. In an interview with Polygon, Microsoft Studios head Phil Spencer says it's a three billion dollar franchise and the "most important entertainment product in the company." Asked if it's the most expensive game Microsoft has made, he says, "Absolutely. Nothing's even close."
But it's also a franchise that has grown up with the User Research department, leaving behind 10 years of legacy data. It's a franchise that spills into other media, like comics and short films, each providing their own data for the group to analyze. It's a game made down the street, so development team members can easily visit the labs. And perhaps most importantly, it's the first game from a new development studio looking to mimic much of a blueprint established by another team over the past 10 years.
One of the most important tasks on Schuh's plate has been helping smooth over that transition. Gathering player data on what feels like old Halo, and what feels different. Seeing what players think of the changes. Assisting with making the new Halo feel enough like the old Halo, and enough like something new.
It's a franchise that spills into other media, like comics and short films, each providing their own data for the group to analyze.
Today, he says, he's just watching for final tweaks and hoping it all works. Though really, he already knows. He's seen the close-to-final survey results, and knows how Halo 4's aggregate user scores compare to those of previous Halo games. He just won't tell us.
Perfecting Bungie's blueprint
At 10:00 a.m. on July 15th, Schuh sets up in "the lounge," a control room where the User Research team and 343 developers watch live tests, for the second day of this two-day weekend test.
Down the hall in the labs, you can hear a hair drop on what looks like daily-shampooed carpet. The walls run neutral colors from floor to ceiling, everyone wears headphones, and the cubicle dividers are large enough so no one sees each other. It's a doctor's office.
"Just turn to the left! To the left! To the left! ... or just fall off that cliff."
The lounge is a frat house. Unevenly glued posters stick to the walls. Stacks of fighting game joysticks spill out from a storage closet in the back. In front sit two 50-inch TVs, each showing feeds from eight Xboxes running the game. Behind sit developers watching how players react. Back at the 343 office across town, more team members tune into live feeds.
Sit in the lounge for more than a few minutes and it feels like watching horse racing or NCAA basketball in a casino gambling pit, with Microsoft employees calling out players by their station initials.
"C04 knows what she's doing."
"B06, what are you doing? Oh my god no ... no! What are you doing?!"
"Don't go over there! Yes, keep doing that! That's what that blip means."
"Just turn to the left! To the left! To the left! ... or just fall off that cliff."
The 343 team members in the room claim they don't place bets on players, impossible as it may seem with all the data that gets thrown around. They're impressed with the diversity in today's test because four of the 18 participants are female. They note that, on average, a game's overall survey score goes up two-tenths to three-tenths of a point when the developers add music.
And they cheer for specific players — in today's test, B03 is the fastest, running towards enemies, using lots of quick melee attacks and at one point flying past enemies in a Banshee [one of the game's aerial vehicles] to save time.
"He's our rabbit," says Schuh.
At one point, a 343 designer notices B03 miss a weapon he wanted him to pick up, so he plans to move its location to make it more obvious. Most tweaks the team calls out are similarly straightforward.
"We didn't have completely working waypoint markers to drive people to the right place."
"A lot of the stuff has been ironed out already," says Schuh. "This is sort of the final cruise. But yeah, we're definitely looking at a bunch of specific small things in each of the missions [today], and we're also just looking at how people are interacting with the Promethians, just to make sure that those are working perfectly."
"One of the things we're going to be looking at is a mission that's fairly late in the game, and it's had some issues with people understanding what the goals are, what they're trying to achieve," says Schuh, keeping things vague to avoid late-game spoilers. "It's one where it's not completely clear. We didn't have completely working waypoint markers to drive people to the right place. And so last time, we saw a lot of people wandering around being lost. We're trying to see if — we put some wav points in; we added audio — to see if that actually solved the problem."
Schuh with Scott Warner, lead designer at 343 Industries
Dan Callan, mission designer at 343 Industries
343 employees watch tests from the lounge, with live video feeds from 16 playtest sessions displayed across two monitors.
Way back when
In the early days of Halo 4 testing, Schuh and team spent less time ironing out kinks, and more time looking at new features.
"One of the first steps was making the game feel like Halo," says 343 executive producer Kiki Wolfkill.
"We did a bunch of iteration and test on different ideas really early on, experimenting with new mechanics — many of which didn't make the cut because of that process," says creative director Josh Holmes.
"One of the first steps was making the game feel like Halo."
One of those was an attempt to evolve the game's jumping system. Historically, Halo's slow player movement has led to what many refer to as "floaty" jumping, and one of the team's initial ideas was to make player movement feel more athletic.
"We played with a higher jump; we played with a smoother jump; we played with being able to go over small jumps without actually needing to physically jump," says Wolfkill. "A lot of things around, 'How can we get some smoother movement there?'"
"The unfortunate side effect of that, though, is it became unpredictable," says Holmes. "Let's say 95% of cases it would do what you expected it to do, but having that 5% be unpredictable [wasn't good enough] ... eventually we just came to the conclusion that 'no matter how hard we work at this, we're always going to have 1% of the time where it's not going to do what the player intends, and that's going to be frustrating to the core user.' And so we ultimately made the decision, after a lot of work and investment in that system, to pull it and to go back to the more conventional jump."
In general, Holmes says, the Halo team values polish and consistency over features that may be exciting but only work 99% of the time.
"I think for Halo in particular, because there's a competitive aspect to it in multiplayer, it's really important to us to keep that consistency," he says. "So players can learn a skill, pick it up, and know that they can apply that skill without the game sort of overriding their expectation. And so philosophically, that is something that we really held very dear."
"We thought we were having this amazing bombastic moment, and it just turned into noise."
That same philosophy came into play when the team set out to balance the game's first mission. Internally known as M10 (where each level follows the convention: M10, M20, etc.), the mission initially threw more at players than they expected from a Halo game.
"We had all these different beats and events that were occurring simultaneously," says Holmes. "So you'd raise the blast shields, you'd see this mysterious planet, but you'd also see a Covenant fleet, and then the planet would open and start pulling you in, but at the same time the Covenant fleet would start boarding the ship. It was super exciting."
But players weren't able to follow what was happening.
"We thought we were having this amazing bombastic moment, and it just turned into noise," says Wolfkill.
So the team broke the scene into pieces, and spread them across different parts of the level, and the survey data came back higher in both 'comprehension' and 'user enjoyment.'
From Schuh's perspective, a job well done.
Kiki Wolfkill, executive producer at 343 Industries
Fire in the hole
The Test subject
One of the playtest participants samples the game in the labs.
After finishing his test, B03 reveals himself to be Steven Peluso. And after Microsoft reps deem him clean cut enough and make him promise not to curse, he agrees to an interview.
Peluso says he almost missed his chance to be here. He remembers getting the call while jogging.
"I'm all really sweaty, and I'm trying to answer my phone, but it's a touch screen so you can't really answer it," he says. "And I'm just like, 'Whatever. I'll call them back later.' And I think it wasn't until like two days later that I called them back. And then, just kind of the questions they asked me, I had a thought that it was Halo, but I didn't necessarily know."
This is the second time Peluso has been called into Microsoft's labs, and his first on Halo. He doesn't live nearby. He drove out for the weekend to stay with a friend because he wanted to play the game.
So what does it feel like on the inside?
"Before I did it, I never really had a problem sitting down and playing games for a long time," he says. "A game would come out the night of release; you'd go home and play it for hours. But then you come here, you sit down, you play it and ... I don't know. It's like seven hours that you sit down and play it. And you kind of forget that when you're playing at home you can get up sometimes, and you go get something to drink, to eat. Here's it's just you and the game."
He says he tries to forget people are watching when he's in the room but feels pressure to play quickly so he can finish the game before the test ends. "You wouldn't go into a movie and leave three-fourths in and be like, 'I'll wait 'til it's out on DVD,'" he says.
He seems relatively understanding of the time he got stuck in the spoiler section. "There was distinctly one point where, from as far as I knew, that's where I was supposed to go. And it took me a few tries before I realized it was just this little small thing, and eventually I got it, and I was just like, 'Huh, well maybe someone should have pointed that out for me.'"
What does he think of the game?
"I think it's much more like Halo 1 than, say, Reach or 3, and to me that's a good thing," he says. "That was my favorite one ... I felt there were more open areas. Granted, there were a lot more closed-in ones, but the open areas have always been what I enjoyed out of the Halo games."
For Schuh and the rest of the team hoping to make the game feel like a natural transition from previous Halo games, this is the kind of thing they want to hear. Something that references old Halo, without being too familiar.
With that, Microsoft gives Peluso his four gratuity items (a year of Xbox Live, an Xbox Live points card, Forza 4 and Alan Wake) and sends him on his way home.
Never ever mention outside
Thanks to non-disclosure agreements, Peluso's Polygon interview marks the only time he's contractually allowed to talk about what he saw before the game releases. Even now, as this story goes live before the game ships, he can't repeat the things we just quoted him saying. Such is Microsoft's security for Halo 4 testing.
Since before 343 Industries existed, Bungie and Microsoft have long been paranoid about keeping Halo's story a secret until each game ships. And the User Research team is similarly security conscious — everyone who enters its labs signs a non-disclosure agreement in the lobby, gets reminded by staff that these things must stay private, and has an individual camera film them while they play.
Even the glass doors to each of the labs have "Studios" logos cut into them in such a way that no one outside can see what's on the screens. Much like in Halo missions, the rooms are positioned so you can see what the designers want you to see.
"Over the 12 years of this group, we have run over 100,000 people through our labs," says Schuh, again reciting the numbers off the top of his head. "We've had seven leaks that we know about."
When things leak, Schuh helps track down the person then hands the name to Microsoft's lawyers. "We're really good about tracking them down, and a lot of the security features that I'm not going to talk about that are subtly here are as a result of things that we've done through tracking them down before."
For Schuh, the frustration comes when leaks make it harder to convince developers to do more user research testing. "Leaking, although it's exciting — it actually ruins the ecology for games, because it makes teams not want to do this type of research, and this is what makes the games better," he says.
Testing for Intent
How halo testing has evolved
Design: Warren Schultheis & Scott Kellum