Data entry, risk management and tacos: Inside Halo 4’s playtest labs

We spend a day in Microsoft's User Research labs to see how the company uses public feedback to polish its most important game.
Jump to
Close

Before going home tonight, Eric Schuh will watch Halo 4's ending 16 times. It will be under less-than romantic circumstances, on blurry six-inch screens with no sound. The code won't be complete. He'll gloss over cutscenes. People will walk in front of him and talk in the background, surgically criticizing the smallest details.

He'll watch players guide series star Master Chief in his fight against new enemies the Prometheans, never once touching a controller.

Schuh doesn't mind. He's done this before.

"We like to keep this as scientific as possible"
Schuh2
Eric Schuh, senior user researcher at Studios User Research

As the Halo lead in Microsoft's Studios User Research and Central Analytics department, he's spent the past two years collecting data on the much-anticipated sci-fi shooter. He and other team members recruit players, sit them in front of Xboxes, then watch, record and analyze everything that happens. They learn what confuses people, and turn their data into advice to the development team on what to babyproof and what to amputate.

Then they speak in numbers. Out of a database of 60,000 volunteers, Schuh says his team can fit 60 in the three rooms in front of him, 18 are playing Halo today and each will answer 884 survey questions before leaving. He's wearing a shirt with "10 years of consolidated research" on the back, in the form of a word cloud showing survey results, and points out $20,000 worth of art on the walls, showing abstract representations of data.

"We like to keep this as scientific as possible," he says. What some companies call "playtests" he calls "studies," what some call "game rooms" he calls "labs," and what some call "players" he calls "participants."

To date, Schuh has run 40 tests on Halo 4's highly secretive single-player campaign - totaling, as he tells it, 1,252 players and $2,750 in pizza.

Today, July 15th, is the last one.

Ready at Midnight

Schuh's road to this point began in 1992. Enrolled in the University of Washington, studying to be a social psychologist, he "almost but not quite" got his PHD when a friend turned him onto a job opening at Microsoft.

Since 2002, he's been in the Xbox group, touching most games that have come through. Collecting and analyzing data, managing risk and limiting player confusion and frustration.

Schuh's done well for himself. After working on Halo: ODST, Schuh graduated into a management role to oversee many of the projects in the User Research department. But when Halo 4 came around — then known under the codename "Midnight" — he stepped down from that role to focus on it.

ASKED IF IT'S THE MOST EXPENSIVE GAME MICROSOFT HAS MADE, HE SAYS, "ABSOLUTELY. NOTHING'S EVEN CLOSE."

That might seem counter intuitive to an outsider, but as he explains it, for someone who works in data collection and user research analysis, it's tough to find a better gig than Halo 4.

"The allure of not eating Top Ramen was pretty high," he says.

On the surface, it's Microsoft's most popular game. In an interview with Polygon, Microsoft Studios head Phil Spencer says it's a three billion dollar franchise and the "most important entertainment product in the company." Asked if it's the most expensive game Microsoft has made, he says, "Absolutely. Nothing's even close."

But it's also a franchise that has grown up with the User Research department, leaving behind 10 years of legacy data. It's a franchise that spills into other media, like comics and short films, each providing their own data for the group to analyze. It's a game made down the street, so development team members can easily visit the labs. And perhaps most importantly, it's the first game from a new development studio looking to mimic much of a blueprint established by another team over the past 10 years.

One of the most important tasks on Schuh's plate has been helping smooth over that transition. Gathering player data on what feels like old Halo, and what feels different. Seeing what players think of the changes. Assisting with making the new Halo feel enough like the old Halo, and enough like something new.

It's a franchise that spills into other media, like comics and short films, each providing their own data for the group to analyze.

Today, he says, he's just watching for final tweaks and hoping it all works. Though really, he already knows. He's seen the close-to-final survey results, and knows how Halo 4's aggregate user scores compare to those of previous Halo games. He just won't tell us.

Darkness_schuh

Perfecting Bungie's blueprint

Newblue

Microsoft's developer transition program for Halo 4 goes back many years.

In August 2007, when the Games User Research department made its first public splash in a Wired magazine cover story, the wheels were already in motion. At the time, developer Bungie Studios used the labs to massage Halo 3 with high tech methods like automatically tracking where players went, what weapons they used and how they died.

In September 2007, Bungie and Microsoft released Halo 3, which according to a Bungie press release was, at the time, "the fastest-selling video game ever, achieving $300 million in global sales its first week."

In October 2007, Bungie announced plans to leave Microsoft and become independent, stating a desire to develop a new franchise.

Microsoft was setting up a long runway to make Halo 4 feel like a natural evolution.

"[Bungie] was weening itself, either consciously or subconsciously, from Halo for a while," says Spencer. "They started a couple secondary projects while Halo 2 or 3 were going on that we didn't complete. Bungie never equalled Halo."

Enter 343 Industries. Around the time of Bungie's departure, Microsoft began to funnel money into a homegrown team that would oversee all parts of the Halo brand — from licensing to fiction to new game development.

Bungie would stick around as a partner until 2010, developing two more Halo games in the transition, but Microsoft was setting up a long runway to make Halo 4 feel like a natural evolution. To hit the hard-to-define target of making the game feel new enough for those wanting something different, and familiar enough for legacy fans.

The kind of thing that one person can't determine alone, but a group of a thousand testers can decide in aggregate.

Five years after Halo 3, we're here to see how things have changed.

Getting comfortable

At 10:00 a.m. on July 15th, Schuh sets up in "the lounge," a control room where the User Research team and 343 developers watch live tests, for the second day of this two-day weekend test.

Down the hall in the labs, you can hear a hair drop on what looks like daily-shampooed carpet. The walls run neutral colors from floor to ceiling, everyone wears headphones, and the cubicle dividers are large enough so no one sees each other. It's a doctor's office.

"Just turn to the left! To the left! To the left! ... or just fall off that cliff."

The lounge is a frat house. Unevenly glued posters stick to the walls. Stacks of fighting game joysticks spill out from a storage closet in the back. In front sit two 50-inch TVs, each showing feeds from eight Xboxes running the game. Behind sit developers watching how players react. Back at the 343 office across town, more team members tune into live feeds.

Sit in the lounge for more than a few minutes and it feels like watching horse racing or NCAA basketball in a casino gambling pit, with Microsoft employees calling out players by their station initials.

"C04 knows what she's doing."

"B06, what are you doing? Oh my god no ... no! What are you doing?!"

"Don't go over there! Yes, keep doing that! That's what that blip means."

"Just turn to the left! To the left! To the left! ... or just fall off that cliff."

The 343 team members in the room claim they don't place bets on players, impossible as it may seem with all the data that gets thrown around. They're impressed with the diversity in today's test because four of the 18 participants are female. They note that, on average, a game's overall survey score goes up two-tenths to three-tenths of a point when the developers add music.

And they cheer for specific players — in today's test, B03 is the fastest, running towards enemies, using lots of quick melee attacks and at one point flying past enemies in a Banshee [one of the game's aerial vehicles] to save time.

"He's our rabbit," says Schuh.

At one point, a 343 designer notices B03 miss a weapon he wanted him to pick up, so he plans to move its location to make it more obvious. Most tweaks the team calls out are similarly straightforward.

"We didn't have completely working waypoint markers to drive people to the right place."

"A lot of the stuff has been ironed out already," says Schuh. "This is sort of the final cruise. But yeah, we're definitely looking at a bunch of specific small things in each of the missions [today], and we're also just looking at how people are interacting with the Promethians, just to make sure that those are working perfectly."

"One of the things we're going to be looking at is a mission that's fairly late in the game, and it's had some issues with people understanding what the goals are, what they're trying to achieve," says Schuh, keeping things vague to avoid late-game spoilers. "It's one where it's not completely clear. We didn't have completely working waypoint markers to drive people to the right place. And so last time, we saw a lot of people wandering around being lost. We're trying to see if — we put some wav points in; we added audio — to see if that actually solved the problem."

Lounge3
Schuh with Scott Warner, lead designer at 343 Industries
Lounge2Dan Callan, mission designer at 343 Industries
Lounge343 employees watch tests from the lounge, with live video feeds from 16 playtest sessions displayed across two monitors.

Way back when

In the early days of Halo 4 testing, Schuh and team spent less time ironing out kinks, and more time looking at new features.

"One of the first steps was making the game feel like Halo," says 343 executive producer Kiki Wolfkill.

"We did a bunch of iteration and test on different ideas really early on, experimenting with new mechanics — many of which didn't make the cut because of that process," says creative director Josh Holmes.

"One of the first steps was making the game feel like Halo."

One of those was an attempt to evolve the game's jumping system. Historically, Halo's slow player movement has led to what many refer to as "floaty" jumping, and one of the team's initial ideas was to make player movement feel more athletic.

"We played with a higher jump; we played with a smoother jump; we played with being able to go over small jumps without actually needing to physically jump," says Wolfkill. "A lot of things around, 'How can we get some smoother movement there?'"

"The unfortunate side effect of that, though, is it became unpredictable," says Holmes. "Let's say 95% of cases it would do what you expected it to do, but having that 5% be unpredictable [wasn't good enough] ... eventually we just came to the conclusion that 'no matter how hard we work at this, we're always going to have 1% of the time where it's not going to do what the player intends, and that's going to be frustrating to the core user.' And so we ultimately made the decision, after a lot of work and investment in that system, to pull it and to go back to the more conventional jump."

"So you'd raise the blast shields, you'd see this mysterious planet, but you'd also see a Covenant fleet, and then the planet would open and start pulling you in, but at the same time the Covenant fleet would start boarding the ship. It was super exciting."
Josh-holmes-bigJosh Holmes, creative director at 343 Industries

In general, Holmes says, the Halo team values polish and consistency over features that may be exciting but only work 99% of the time.

"I think for Halo in particular, because there's a competitive aspect to it in multiplayer, it's really important to us to keep that consistency," he says. "So players can learn a skill, pick it up, and know that they can apply that skill without the game sort of overriding their expectation. And so philosophically, that is something that we really held very dear."

"We thought we were having this amazing bombastic moment, and it just turned into noise."

That same philosophy came into play when the team set out to balance the game's first mission. Internally known as M10 (where each level follows the convention: M10, M20, etc.), the mission initially threw more at players than they expected from a Halo game.

"We had all these different beats and events that were occurring simultaneously," says Holmes. "So you'd raise the blast shields, you'd see this mysterious planet, but you'd also see a Covenant fleet, and then the planet would open and start pulling you in, but at the same time the Covenant fleet would start boarding the ship. It was super exciting."

But players weren't able to follow what was happening.

"We thought we were having this amazing bombastic moment, and it just turned into noise," says Wolfkill.

So the team broke the scene into pieces, and spread them across different parts of the level, and the survey data came back higher in both 'comprehension' and 'user enjoyment.'

From Schuh's perspective, a job well done.

Wolfkill
Kiki Wolfkill, executive producer at 343 Industries

When lines blur

Lounging

After working in user research for 20 years, Schuh tends to blur the lines between where his job stops and life begins.

"It is just part of my DNA at this point," he says.

Similar to how a critic often finds themselves critical of the world at large, he recently caught himself off guard when he found himself analyzing the usability of food. Looking at a taco, he noticed that its shell wasn't holding up.

"Why does the crunchy shell shatter and drop food all over your crotch," he wondered.

"Why does the crunchy shell shatter and drop food all over your crotch?"

The usability report would run all the details — how steam and oil make the shell soggy, how a wrapper keeps the ingredients contained, how certain items perform better in the restaurant and others perform better at the drive-through window.

So he's a fan of Taco Bell's gorditas, then, with a soft tortilla backing up a fragile crunchy one? A bit bulkier, arguably tastier, but easier to eat.

"Exactly. Yeah, that's great usability. Good fix for that problem."

Multiplayer playtesting

Longbow-ctf-trans

While our story focuses on Microsoft using the public to help polish Halo 4's single-player campaign, the company has other testing processes in place to handle different games and modes, to look for bugs in a game's code, to watch a single person react to a game live through two-way mirrors, to adjust physical controls for Kinect games and to playtest games internally rather than relying on the public.

Longbow-ctf

One of these, which Schuh also works on, is Halo multiplayer testing. Much like the campaign playtests, this involves bringing in the public and seeing what paths they take, if they understand the maps, and how they die. Schuh and his team then consolidate the multiplayer data and create reports and heat maps (as seen above) for the developers to understand how players use the maps. Schuh says it's usually a good sign to see a handful of main paths players take through a map rather than seeing them run all over the place, confused as to where they should go.

Fire in the hole

"It looks like he's feeling really compelled to shoot. Is there any reason to shoot those things, or are they just there for flair?"
Blueprint23

Around 2:00 on July 15th, Schuh and the 343 team members run into an issue that gets a lot of attention in the lounge. The same one Schuh mentioned he'd be looking at earlier in the day.

Over the course of the morning, the group identified plenty of minor bumps, most of which they say are easy fixes. At the top of a ramp, a player tried to throw a grenade through translucent glass, signaling a designer to change its opacity. At the end of a mission, another player got lost in thick fog, so the team plans to thin it down. Most of these tweaks have to do with changing the lighting, adding waypoint markers to point players in the right direction, and moving weapon locations.

The one that gets the most attention comes in a part of the game many would consider a spoiler, so we'll leave out specifics. But late in the game, player B03 — the player the team noticed rushing through the game earlier — gets lost and doesn't know where to go to reach the end of a mission.

"Did B03 turn around again?" asks one developer.

"Did he not go in there?" asks another

"He didn't figure out the wall."

"Is he back there again?"

"Did he die? Wait, what just happened? He just popped. Did he teleport forward?"

"He's died a bunch of times."

Then the 343 comments calm and shift towards solutions. A designer notices that when B03 turns around, he runs into oncoming fire, which isn't supposed to happen — one of Halo's long held subtle design traits is that if a player sees someone shooting at them, they know they're going in the right direction.

"It looks like he's feeling really compelled to shoot. Is there any reason to shoot those things, or are they just there for flair?" asks a developer.

"They're for flair."

"They're there because everyone likes to shoot them."

Another designer notices that the lighting isn't properly pointing B03 to the exit, and doesn't dim to subtly suggest he's going the wrong way once he gets lost. Something that the team would only notice once a player finds themselves lost, and something that most players would never need. But it hits the 99% barrier that Holmes and the team want to avoid.

"Once the [geometry] got more complicated in there, it got harder to navigate," says a team member. "And then lighting is also an issue."

Ultimately, Schuh says, this issue is as fixable as most of the others they've discovered, but it tends to stand out both because it happens during an important moment in the game and because it lasts longer than most of the other hiccups in the test.

"I was feeling for [him] as he was going through it, and it wasn't his fault," says Schuh after the fact. "We just need to draw players' attention to the place that they need to go, which we'll be able to do with lighting, and then sort of turn off the lights behind him so that he knows, 'Hey you're going the wrong way when you're going backwards,' and it'll be all good."

The Test subject

Tester
One of the playtest participants samples the game in the labs.

After finishing his test, B03 reveals himself to be Steven Peluso. And after Microsoft reps deem him clean cut enough and make him promise not to curse, he agrees to an interview.

Peluso says he almost missed his chance to be here. He remembers getting the call while jogging.

"I'm all really sweaty, and I'm trying to answer my phone, but it's a touch screen so you can't really answer it," he says. "And I'm just like, 'Whatever. I'll call them back later.' And I think it wasn't until like two days later that I called them back. And then, just kind of the questions they asked me, I had a thought that it was Halo, but I didn't necessarily know."

This is the second time Peluso has been called into Microsoft's labs, and his first on Halo. He doesn't live nearby. He drove out for the weekend to stay with a friend because he wanted to play the game.

So what does it feel like on the inside?

"Before I did it, I never really had a problem sitting down and playing games for a long time," he says. "A game would come out the night of release; you'd go home and play it for hours. But then you come here, you sit down, you play it and ... I don't know. It's like seven hours that you sit down and play it. And you kind of forget that when you're playing at home you can get up sometimes, and you go get something to drink, to eat. Here's it's just you and the game."

He says he tries to forget people are watching when he's in the room but feels pressure to play quickly so he can finish the game before the test ends. "You wouldn't go into a movie and leave three-fourths in and be like, 'I'll wait 'til it's out on DVD,'" he says.

He seems relatively understanding of the time he got stuck in the spoiler section. "There was distinctly one point where, from as far as I knew, that's where I was supposed to go. And it took me a few tries before I realized it was just this little small thing, and eventually I got it, and I was just like, 'Huh, well maybe someone should have pointed that out for me.'"

What does he think of the game?

"I think it's much more like Halo 1 than, say, Reach or 3, and to me that's a good thing," he says. "That was my favorite one ... I felt there were more open areas. Granted, there were a lot more closed-in ones, but the open areas have always been what I enjoyed out of the Halo games."

For Schuh and the rest of the team hoping to make the game feel like a natural transition from previous Halo games, this is the kind of thing they want to hear. Something that references old Halo, without being too familiar.

With that, Microsoft gives Peluso his four gratuity items (a year of Xbox Live, an Xbox Live points card, Forza 4 and Alan Wake) and sends him on his way home.

Diescumhalo

Never ever mention outside

Thanks to non-disclosure agreements, Peluso's Polygon interview marks the only time he's contractually allowed to talk about what he saw before the game releases. Even now, as this story goes live before the game ships, he can't repeat the things we just quoted him saying. Such is Microsoft's security for Halo 4 testing.

Since before 343 Industries existed, Bungie and Microsoft have long been paranoid about keeping Halo's story a secret until each game ships. And the User Research team is similarly security conscious — everyone who enters its labs signs a non-disclosure agreement in the lobby, gets reminded by staff that these things must stay private, and has an individual camera film them while they play.

Even the glass doors to each of the labs have "Studios" logos cut into them in such a way that no one outside can see what's on the screens. Much like in Halo missions, the rooms are positioned so you can see what the designers want you to see.

"Over the 12 years of this group, we have run over 100,000 people through our labs," says Schuh, again reciting the numbers off the top of his head. "We've had seven leaks that we know about."

When things leak, Schuh helps track down the person then hands the name to Microsoft's lawyers. "We're really good about tracking them down, and a lot of the security features that I'm not going to talk about that are subtly here are as a result of things that we've done through tracking them down before."

For Schuh, the frustration comes when leaks make it harder to convince developers to do more user research testing. "Leaking, although it's exciting — it actually ruins the ecology for games, because it makes teams not want to do this type of research, and this is what makes the games better," he says.

Testing for Intent

Ultimately, according to Schuh, it's up to the developer how much they use the feedback from the User Research group. And 343, he says, is one of the most active teams involved.
Outsidehalo

When Schuh talks about User Research testing making games better, he says it with confidence. There's no doubt in his mind that it significantly improves games to use data to smooth out experiences.

Not every experience wants to be smooth, however. Creative types often cite a desire not to get too much outside input on their ideas. The directors of the recent Indie Game: The Movie documentary bookended their film with a quote from Braid designer Jonathan Blow — the only central quote to appear in the movie twice — highlighting this divide.

"[Big companies] create highly polished things that serve as large of an audience as possible," he said. "The way that you do that is by filing off all the bumps on something; if there's a sharp corner, you make sure that's not going to hurt anybody if they bump into it or whatever. That creation of this highly glossy commercial product is the opposite of making something personal."

Blow's comments refer less literally than they do creatively, and few would claim that Halo is trying to be a game about one person's vision, but his overall point carries over — too much external input can sometimes push developers into areas they would prefer not to go. Changing the geometry of a level to make it more user friendly, for instance, may make it less attractive. Like the thicker taco shell, fixing a problem doesn't always create a better product.

Schuh and the designers at 343 make a clear distinction here, saying they're trying to perfect a creative vision rather than change its substance, noting that ultimately the design choices come down to the design team. They use the phrase "confirming intent" to point out that this process is not about challenging creative ideas so much as discovering whether those ideas resonate.

"It comes down to what the intent is," says Holmes. "There are moments where you want the player to feel that sense of being a little bit lost and disoriented. And then it's more about finding the balance of, 'How long do you want them to maintain that sensation before giving them the information that they need to drive forward?' So a lot of it just comes down to testing against intent. There's no magic set of metrics where you're like, 'OK, you've just got to tick off these boxes and then you've got yourself a great experience and a great level.' You go in with a very clear intention, but then you're testing to see that you are actually achieving that intention."

Ultimately, according to Schuh, it's up to the developer how much they use the feedback from the User Research group. And 343, he says, is one of the most active teams involved.

"This was my first experience really working with this sophisticated user research process and group, and it's been a godsend," says Holmes. "I've worked in the past at companies that put a lot of effort into testing at the concept level — doing focus testing, and trying to figure out, 'OK, is this idea resonating?' But not really having a formalized process for testing how players experience the game and whether you're getting the clarity you want."

"One of the big reasons I came to work here was this department," says lead designer Scott Warner. "I'm not joking when I say, top three reasons I came to 343, one was user research.

"A lot of people would say, 'Well we'll watch people play and see how things go,' but there [are] ways of doing this [that work better than others]. And Microsoft has a really long history of doing this kind of testing, not only through games but also obviously through applications and [everything else they produce]."

How halo testing has evolved

Oldhalo


For Schuh, it's easy to chart the evolution of the Xbox User Research approach alongside the evolution of Halo.

In the Halo 1 days, the User Research team had three or four researchers each covering approximately six games, whereas now the team is — depending how you count the numbers — roughly 24-42 people and Halo 4 is Schuh's full time job.

"Because of that, in Halo 1, we were only able to do two-hour game slices," Schuh says. "We didn't look at the entire game. So yeah, the first two hours of Halo had our touch and had our love, and beyond that — where some levels might be a little confusing — those were beyond what we looked at."

Halo 2 marked the first push into Halo multiplayer user testing, which Schuh says led to observation challenges due to a lack of consistency.

"I think that the biggest difference is that campaign, in many ways, is more complex, because you've got all sorts of different ways of dying," he says. "[In multiplayer] I know that another player is killing them. And you could fall to your death, and we do have that data that's represented. But there are so many different things that can kill you in the campaign. And [in multiplayer we have to look for], 'Where was the enemy that actually shot you? Was he up on this rock?'"

That led to the most high tech step forward, where the User Research team worked with developers to "instrument" Halo 3 — meaning, they set up a system to automatically record and catalog player data, which they could then use to create heat maps showing how players use the game's maps.

"So whenever someone dies, what weapon they had in their hands, what enemy killed them, what weapon did the enemy use, where were they, what was their health before they started this encounter — we take all of this information, both the attitude data from the surveys, and the instrumentation data from the game, and we put them together to get an understanding," says Schuh. "Because otherwise we would see, 'Hey someone died 10 times, but I don't know if 10 times is good or bad.' The only way I can find that out is by getting the attitudes.

For Halo: ODST, the User Research team folded in the game's Firefight mode — which pits a group of players against waves of enemies — presenting new consistency challenges. And when Halo: Reach came around, the User Research team expanded its competitive multiplayer testing scope beyond the Slayer deathmatch mode into other objective-based modes.

With Halo 4, the team has attempted to test everything in the game. "We have looked at pretty much every multiplayer map and mode, all of Spartan Ops, and the entire campaign," says Schuh, "and all the new additions to the Halo sandbox."

In recent years, the team has also evolved its role to study "broader business trends," Xbox Live data, Nielsen TV ratings, and other data that Schuh can't mention to create an "ecosystem" to help analyze games.

"We've got lots of different data streams," says Schuh. "So we do have our playtest data. We have our usability data. But there are other data streams that are out there as well — what games are most popular, when are they playing, what modes are they playing ... what shows are people watching? Everything."

The future of user research testing

While Microsoft's User Research team has evolved in recent years into complex data comparisons, other companies have attempted more high tech — if at the moment less practical — approaches to analyzing players.

At the 2011 Game Developers Conference, Valve's Mike Ambinder showed experiments on how developers could record "skin conductance levels" via a blood pressure monitor on a player's hand, and track players' eyes to learn where they plan to move or look next. His experiments were based around the idea of using this data to change a game's design in real time, but apply to the same kinds of user analysis that could theoretically benefit a game like Halo.

However, in our research, we found that while a handful of companies dabble in this kind of testing, few actively use it on games in production. Facial recording and analysis, especially amongst mobile developers and publishers, is more common.

Schuh says he's interested in chasing the most practical results, rather than the most high tech approach. While each station in Microsoft's labs includes a camera filming players' facial reactions, for example, he says he doesn't spend a lot of time interpreting those reactions for Halo because it's a challenge to convert them into consistent data.

"We've done some research into that, and the thing that we've found is, as sexy as it seems as it would be, in order to get people's true attitudes by looking at their faces [you need consistency]," he says. "People have a lot of variety about how they actually interact. Some people look very deadpan when they're really enjoying things, and some people are laughing because they think something is ridiculously bad ... We're not just here about collecting information to understand things. We're here about making the game better by having actionable data, and knowing someone smiled at four minutes and 38 seconds into a mission doesn't help us out a lot."

For the future of User Research testing, Schuh is more interested in spending more time away from the office and getting into real world scenarios.

"I'm really interested in looking at stuff outside of our labs," he says. "We do do some research outside of our labs, but I am really interested in what happens when you've got a smart device in your hand that has photographic and voice capture capabilities, and how you can combine that with the type of research that we do, to understand better how people are using technology and games."

6:00 p.m.

Schuh looks relieved. Nothing big went wrong. Which is what he expected would happen. Which is what usually happens at this point in the cycle. But also which was necessary for the game to comfortably hit its November 6th release date.
Pizzaboxes2

By the end of the day, everyone in the Halo 4 test has finished the campaign.

Without audio in many cutscenes, it's difficult to say whether they understood exactly what happened in the story, but they got their gratuity and Schuh's team got its data.

As Schuh and team clean up the Papa John's pizza boxes and plates scattered around the lounge, Schuh looks relieved. Nothing big went wrong. Which is what he expected would happen. Which is what usually happens at this point in the cycle. But also which was necessary for the game to comfortably hit its November 6th release date.

Just a bit ago, he gave a quick mission-by-mission preview to the 343 team of what his report on this test will include. And now he's ready to call it a day and head home for the last few hours left of his weekend.

"It's been a long journey," he says. "It's exciting to get to this point — long day, long weekend, but exciting."

Schuh's job isn't done, but as he says, it's "winding down." Next on his plate: a report summarizing this campaign test, meetings with 343 team members to discuss what the data means, a final pre-release multiplayer test, a "formality" look at the campaign to check the team's work, and ongoing support once the game is released.

As companies have evolved to treat certain games more like services than products, so has Microsoft's User Research group. On a high profile title like Halo — especially one with a significant multiplayer mode and an episodic series of "Spartan Ops" missions planned — that means Schuh's heavy lifting is done for the moment, but his job on Halo 4 will continue into next year.

"We're keeping it alive," he says, "so there is always an understanding of how players are playing our maps. They're always going to feed into the next version of the game. They're going to help us tune for any title updates and tuning passes that we do. So we don't stop once the game is over. This is just the start of the next phase."

Ask when he'll be 100% done with Halo 4, and Schuh can't commit to an answer — though it's hard to say if he's being cagey, or legitimately doesn't know, or if there's even much of a break before the next Halo comes around.

"We stop when ... it just keeps going," he says. "That's the great thing about this franchise." Babykayak

Design: Warren Schultheis & Scott Kellum

X
Log In Sign Up

forgot?
Log In Sign Up

Forgot password?

We'll email you a reset link.

If you signed up using a 3rd party account like Facebook or Twitter, please login with it instead.

Forgot password?

Try another email?

Almost done,

Spinner

Authenticating

Great!

Choose an available username to complete sign up.

In order to provide our users with a better overall experience, we ask for more information from Facebook when using it to login so that we can learn more about our audience and provide you with the best possible experience. We do not store specific user data and the sharing of it is not required to login with Facebook.

tracking_pixel_5353_tracker