In late 2013, Polygon published “The birth of Xbox Live,” an extensive feature covering the origins of Microsoft's online service up through the release of Halo 2. Now we're back with part two below, covering Live's evolution through the Xbox 360 years.
They were gathered together in a hotel to play a game. J. Allard, Robbie Bach and a handful of associates. The many heads of Xbox. It wasn't going to be a fun game, and in the end, no matter which team won, they were all, or so they thought, going to lose.
It was 2005, some time before the launch of the Xbox 360. The game was a "war game" of sorts: They were role-playing the economics and marketing of launching major video game consoles. There were two teams. One team was playing Microsoft; the other, Sony Computer Entertainment. The goal for each team: Sell more consoles than the other guys.
Microsoft was about to launch its Xbox 360. Sony was planning its third PlayStation console. The two teams in Microsoft’s executive game pretended to make announcements. They prepared for E3 reveals. They planned their marketing strategies, release dates and launched games.
The rules were set by a consulting firm, which also supplied the publicly (and semipublicly) available data on both companies. The firm had built its own simulation, from its research, on what was likely to happen in the coming console war. The men on each team recreated that simulation, adjusting variables, changing strategies, trying to do better than the estimates. They believed the fate of their endeavor — and, really, their careers — depended on it. Because, in the original estimates, they lost. Again.
At the time, Microsoft was decidedly second fiddle in the console war between the original Xbox (some at Microsoft now call it "Xbox Prime") and PlayStation 2. Microsoft knew it was playing catch-up in a market it had only recently decided to enter. But what it didn't know, yet, was how hard and how fast it would have to chase after Sony. That's what this game was designed to show them: how far they had left to go to reach the top. What could they do with what they had in order to sell more units than Sony? How could they win?
The one advantage Microsoft had going into this war, and into this game, was Xbox Live, the still fledgling online gaming service connecting Xbox Prime over broadband internet. Sony's own network wasn't anything like it, and the powerful Japanese giant was, itself, playing catch-up in the field of connected gaming. Microsoft believed, based on Allard's predictions in the mid-1990s, that online was the future. So it was doubling down: Microsoft's new console would have Xbox Live built right in, and the service itself was getting an expensive overhaul, front to back.
It would be another huge (and expensive) gamble for the people at Microsoft's Millennium E campus. But even within Xbox, there were questions about how far out on the limb they could go without falling.
"Xbox Live was still considered vaporware. It was like, ‘What do you have? Maybe a million users?’"
"Xbox Live was still considered vaporware," says then Program Manager Jerry Hook. (He's now an executive producer at 343 Industries, working on future Halo games.) "It was like, 'What do you have? Maybe a million users?' ... It wasn't proving yet for mainstream consumers. It wasn't proving yet for publishers."
Even after the successes of Xbox Prime and Xbox Live, some within Microsoft were asking questions about whether the Xbox experiment would, in the end, be just a fun write-off or eventually become a business. Now it was time to get serious or go home.
At the war game, Xbox Senior VP J. Allard played the role of Sony Computer Entertainment head Ken Kutaragi, squaring off against Robbie Bach as head of Xbox. Each team made their announcements, adjusted the specs of their boxes, and dealt with the repercussions of their decisions on things like production pipelines, timing and cost. They ran the game in every configuration imaginable, multiple times. They switched roles, changed teams. At the end of each session the consulting firm would give them a report on how many units each team could expect to sell. Each time, there was a winner and a loser. Each time, that winner was Sony and that loser Microsoft. Time and again, no matter what variables changed, the new PlayStation pummeled the Xbox 360.
The result seemed clear: Microsoft was going to lose this war.
Xbox's key executives left the game believing Xbox 360 would get beat. Their only chance, they had concluded, would be if Sony literally bungled every single step. The PS3 would have to suffer the most conceivably flawed product launch of any video game console in history in order for it to sell fewer units than the Xbox 360. It would have to be priced higher, ship later and come to market with fewer games than the Xbox 360. And it would have to be marketed so poorly that two generations of dedicated Sony gamers would consider jumping ship for Microsoft's console.
The only way to win, Xbox believed, was if Sony didn't play. At the time, nobody believed such a thing was possible. They were wrong.
"We actually have a completely redundant facility from which we can do everything we can do here, in another building on another continent. This building is ... on multiple power grids. We have huge diesel generators. We have 72 hours of diesel in a tank out in the parking lot." — Eric Neustadter
In the experience of the millions of people who use it, Xbox Live is an intangible thing. It's a connection; it's a pipeline; it's a service. It plugs into or connects wirelessly to a box or a phone. It is both there and not. For the people who help run Xbox Live, it's a building.
The Xbox Operations Center is a large room filled with computers and screens, and like almost everything else at Xbox, it has an acronym with an X in it. The "XOC," as they call it, looks like small movie theater with tiered rows of desks all facing a large wall adorned with a dizzying array of screens. It looks a lot like NASA Mission Control. The people sitting at the desks each have three screens, one for the Microsoft corporate network, one for the Xbox network and another for various other functions. There are around 40 or 50 of these workstations, and the people who sit at them work in shifts and have varying specialties.
One person's job is to do nothing but watch "the clock," a digital readout at the top of the big wall of screens which starts running only when there's a problem. Usually, it reads zero. If it reads anything other than zero, that means something on Xbox Live has gone wrong. And this person, this clock-watcher, does nothing but watch the clock count upward and let everyone else know what it reads, so everyone else can focus on solving the problem.
It's late 2013, on a Wednesday. I've been invited to Microsoft's campus to hear the story of how Xbox Live was rebuilt, twice, between the launch of Halo 2 in 2004 and the rollout of the New Xbox Experience update in 2008. I'm getting a tour of the XOC so that I might understand just how big an endeavor this really is — running Xbox Live. On this day, the XOC is full of people because it's an "overlap" day. Everyone's shifts have been extended so they can hold team meetings and compare notes. Their meetings, most likely, are about Xbox One.
Microsoft's new console has just launched. Most of the people working in the XOC have been working or on call for weeks, staring at the screens, waiting for problems to arise and then solving them. One such problem occurred on the eve of launch day, just before midnight in the U.K. Activity that is carefully monitored changed, server farms stopped responding, traffic on the network dipped and alarms for all of these indicators and more went off, one after another. The clock started.
The people sitting at the desks opened binders full of checklists and data. The investigation began, and then slowly, methodically, the problem was chased to its source. Everything happened in sequence, in a carefully planned, tested and rehearsed ballet of system checks and software analysis. The XOC team checked the internal WTFC (What the Fuck Changed) website — a running list of every change made to any device or service at Microsoft that could impact Xbox Live — to see what could have caused the problem. The problem was rapidly identified and slowly, step by step, resolved. The network came back online and launch day resumed. Total downtime: about an hour.
What happened to cause Xbox Live to crash on that day is still a closely guarded secret. What Microsoft is willing to disclose is that someone at the company did something, somewhere, that broke it. Beyond that, no one will say. The fact is, it could have been anything. What started as an experiment by a small team at Microsoft's Millennium E campus as a gaming service is now a global entertainment network driving not just games, but video, music, email and more to millions of devices around the world. Phones, music players, Xboxes, computers and tablets all running some version of Microsoft's operating system now plug directly into Xbox Live. This pioneering project from a group of renegade engineer gamers is now the digital thread connecting everything Microsoft owns.
"He's got this amazing knowledge and talent and passion for the core gamer. ... Eric still, today, he'll start in a staff meeting just going crazy about this complete corner case of the top 1 percent of gaming. 'This is a huge problem with the service that we need to solve today!' ... The balance between me and Eric, with his knowledge of the core gamer and what developers need, and my background in running large-scale services, I think has served us very well." — Derek Ingalls
At any given time, never too far from the XOC, you'll find Eric Neustadter, the man who laid the foundation.
"We’re sitting on the floor ... going over, literally, how many new servers we’re going to buy to run Xbox Live because of the scale Halo 2 will bring to us."
"It was early 2004," Neustadter says of the time. "It was myself and Pete Parsons and my boss at the time, Ben Kilgore. We're sitting on the floor of Ben's office going over, literally, how many new servers we're going to buy to run Xbox Live because of the scale Halo 2 will bring to us, because no game, before or since, ever grew our user base that much, instantly. ... Things were a lot different back then."
Neustadter, also known by his Xbox Live gamertag "E," was one of the first handful of Xbox Live employees. He is revered by fellow Xbox teammates and, to date, is one of the most well-recognized employees by players. At the beginning of Live, when a service to connect millions of gaming consoles was still just a theory, the entirety of Xbox Live was running on desktop PCs under its developers' desks. Neustadter had a direct hand in building that early testbed into a reliable and functioning server farm. Since the launch of Live in 2002, he's been the go-to person for everything about the service.
Today Neustadter works as the operations architect, which basically means making sure upcoming games add new features and don't break the service. And he remains on call if something does break. He formerly worked at another MIcrosoft division, in another state entirely. he believes he got his job at Xbox because he’s not only a brilliant systems engineer, but also a gamer.
"I'm up here [in Redmond] for my interview," Neustadter says. "I'm talking to one of the early team members about how I game online currently. And so we wound up talking about Age of Empires. I mentioned that one of the things my teammate and I had discovered recently, and this was in 2001 [so it was] really rare, was voice chat. ... We discovered this app called Battlefield Communicator, which was the first DirectX voice chat app. It was like 30 bucks a person. ... I'm explaining this and the guy I'm interviewing with is looking at me kinda funny. He's like, 'Are you serious?' I'm like, 'Yeah ...' He's like, 'Hang on a second.' He dives into his computer and pulls up an Excel spreadsheet. It turns out that he was one of the two developers that wrote that program."
Neustadter also had another advantage: He was one of the few employees at Microsoft who understood both games and running services that had to be stable 24 hours a day, seven days a week. His previous job was consulting with Fortune 500 companies running Microsoft enterprise solutions, an experience that made him invaluable to the fledgling Live team. When Neustadter joined Team Xbox, he was its only network and security engineer.
"Back then, we were running on Windows 2000 Dell rack mount hardware that is so old now it's laughable," Neustadter says. "Really old Cisco gear."
For the next box, however, that would have to change. On Xbox Prime, Live was a service for hardcore gamers. Microsoft had bigger plans for its next console: It would be a device for every consumer, and it would be built from the ground up to not only support Xbox Live, but to rely on it.
There were also plans within Microsoft's sprawling campus for new devices; phones, music players and more, all using Live. The next Xbox would be a Trojan horse, introducing the next generation of gamers to the idea of an "always-on" experience, and to support that vision, Live itself would have to evolve. It would have to become stable 24 hours a day, seven days a week and never go down, for any reason — ever. Which meant Neustadter was about to get company.
"There's something we called the Book of Xenon, which was the code name for the 360. [It] was this big internal document that was basically the plan. We did the exact same process for Xbox One. We had very similar principles. We were very clear about what we were trying to accomplish. 'Where was our innovation going to be, and where was our innovation not going to be?'" — Ben Kilgore
Planning for the Xbox 360 began shortly after the launch of Xbox Live in 2002. Microsoft, the lumbering business software and operating system giant, had proven it could create and release a competitive gaming product in Xbox Prime. And then it followed that product with a robust and capable online gaming service in Xbox Live. The next phase was to merge the two into a single product built from the ground up for online play. It was codenamed Xenon, and it would be released in 2005.
"[Xenon] was the Live console. That was the way I always thought of it."
"[Xenon] was the Live console," says then Unit Manager Ben Kilgore (he's now CTO at Blizzard). "That was the way I always thought of it. People asked, 'What's the features?' It's Live."
Xbox Live had created the opportunity for game makers to tap into features that had, more or less, already been built by Xbox, but the implementation of those features was up to each game developer. Things like finding friends, joining a game or sending a message were all dependent on how (or if) a game developer chose to add that code into their games. The end result was that most games did even basic things differently, causing confusion for gamers.
"[T]here were often times where you'd say that the best way to tell someone how to get to their friends list was to press the eject button," says then Xbox GM Marc Whitten, "because then you could at least show them the one way to do it on the dashboard. Which was obviously the last thing we wanted to happen."
The "big idea" for Xenon was to integrate all of Live's functionality directly into the box. Instead of relying on game makers to implement the code, it would already be there. One menu, one system, one service that would be consistent and reliable across every game. And to get to that service menu, there'd be one button: a convex, X-emblazoned bubble built right into the game controller that would, no matter what the Xbox was doing at the time, go directly to the Xbox Action Menu, or "XAM." The Xbox Team called this innovation "Live Aware."
Every game, service or feature on Xenon would be Live Aware. By pressing the button and activating the XAM, users could access all of the features that would already be there, running under the game, on Xenon, just waiting to be needed. The friends list would always be there, accessible from within any game. Systems settings and more: already there, seamlessly accessible. Live Aware.
"It's a huge load off of game developers, because they didn't have to figure out how to build out this friends stuff," says Whitten. "If they wanted to do something interesting, if they were going to take your friends list and do something new with it in games, they were allowed to do it. They could use it and go off and be creative, but they weren't stuck recreating the core experience of Xbox Live."
But there was another motive behind Live Aware besides making life easier for game developers: the health of Xbox Live itself. Microsoft had been carefully upgrading and maintaining the original Xbox Live infrastructure since the service's launch in 2002, and in the process, it had noticed a curious thing: The system wasn't always busy. Specifically, there were peaks and valleys in the usage of the service, when gamers had time to play or when they were off doing other things. While the system downtime wasn't harmful, it was wasteful. But the peaks, in particular, were worrying. A sudden onrush of activity, say, at the launch of a new game, could have disastrous consequences for the stability of Xbox Live.
"There have been suburbs in major metropolitan areas where in the first 60 seconds of halftime during the Super Bowl, the plumbing infrastructure has collapsed from all of the toilets flushing."
"Pre-[Xenon] we had what was called a very spiky service," says then Executive Producer Jeff Henshaw (now the group program manager at Xbox). "You'd have relative periods of inactivity, and then you would have very active users in very high numbers during comparatively narrow periods of time. That's an example of where you learn things like, Super Bowl halftime. ... There have been suburbs in major metropolitan areas where in the first 60 seconds of halftime during the Super Bowl, the plumbing infrastructure has collapsed from all of the toilets flushing."
Live Aware was intended to smooth those spikes. The system would always be monitoring the activity of each box, sending notices back to the XOC when things went wrong or when they needed to prepare for more data, say, when a game was loaded. And the new version of Xbox Live would add additional features to fill in some of the valleys. Each Xenon box connected to the service would be continuously active, maintaining steady flow of traffic, as opposed to Xbox Prime's, which would lay dormant for weeks, sometimes months, and then flood the service with traffic all at once.
Xenon's designers used the terms "hyper-connected console," and "always-connected console" to promote the next box, both internally and externally. Xenon's "inherent" connection to Xbox Live would "light up your experiences," making Live the "heartbeat of Xenon." And all of these buzzy catchphrases and more made their way into the collection of documents, illustrations, presentations and charts they called The Book of Xenon.
"The Book of Xenon was ... 'How do we keep everyone on the same page?'" says Hook. "'What are the priorities? What [are] the end-to-end consumer scenarios? How do you talk through that?' ... To me, the Book of Xenon was basically our bible."
The Book of Xenon was created from a series of white papers and concept presentations about what the user experience for Xenon would be. How games would load, what it would look like to sign on to Live and use the various services. Everything was imagined and designed so that the team building the box, the team promoting the new device to game makers and the management team talking upward to Microsoft would all be speaking the same language and be working on the same things at the same times. It was, in effect, the heartbeat of the heartbeat.
Collected from years of research, data and ideas, and based on Allard's theories, The Book of Xenon pointed the way to how an "always-connected" console would work. And for the next few years, the team at Millennium E worked steadily toward creating the box that would deliver that experience. But not everyone was convinced.
As a game console, first and foremost, the success of Xenon would depend on it having games that gamers wanted to play on it, and some of Xenon's most vocal opponents were the very developers who would be making those games.
Project Orange, Microsoft's plan in the early days of Live to sell games digitally
While reporting on the first part of this story in 2013, I spoke with Xbox visionary J. Allard about what he called "Project Orange," which was the Xbox team's idea to sell games over the internet, without a disc. This type of digital sale became a key selling point for Microsoft's Xbox 360 follow-up, the Xbox One, but a consumer and retailer backlash to certain components of that plan forced Microsoft to rapidly change gears and scrap the plan to release the box without a disc drive at all.
While it is possible today to buy most games purely digitally, a true digital-only console may be another generation away or more. But Microsoft was already planning for it a decade ago. In early 2014 (just after the Xbox One shipped) I spoke with Xbox Partner Director of Development Boyd Multerer about exactly what Project Orange was about back in 2004.
"It was me and another guy ... we sat down and started talking about, 'Well, why do the licenses have to be on discs?' Right? That leads to all kinds of questions in your mind.
"In gaming consoles, the license to play the game is strongly implied by ownership of the physical media. The arrangement of the bits. ... There's cryptographic things baked into those discs that prove it's a real disc and you own a license. But really, if Live is there, why does the license have to be on the disc? Couldn't it be in the cloud?
"We came up with a program called Orange, which was a proposal on how you could sell games. At the time, it was how you could sell games on [Xenon] that didn't have a license on the disc. You had the license in the cloud. Maybe you used a disc to transport bits around, but who cares about the disc then? It's just a big packet. It's a packet with a lot of bandwidth and a lot of latency, right? And you separate out the license. That didn't fly at the time.
"I think we suggested it too late for [Xenon]. There wasn't enough time in the program to make it happen. But it is the root of the licensing model now on Xbox One. Sometimes you gotta wait a long time for these things to happen. But now we can do things like, you can have a license that's in the cloud, and you can download the bits, or maybe you go the traditional route where you buy a disc and the license is baked on the disc. It opens up your options for how you deliver content. That maps nicely into other kinds of content. It doesn't just have to be a game, although a game is nice because there's a ton of virtual scarcity inside of it. You need the interactivity.
"That was probably when we started thinking along those lines. But once you start going down that path, you realize that licensing is mixed in with content delivery. Once you're going down that path, you realize there's all kinds of content out there that could be delivered in different ways. It lets you think about bit delivery separate from license delivery. You get the bits down any way you can, and you get the license down in more of a controlled manner, but that's OK because the number of bits you're talking about there is much smaller. ... Then that leads to other kinds of media.
"What does it mean to get a movie? What does it mean to get music? What does it mean to have licenses to those things? Those are a little simpler. The game is by far a more complex model. But it leads you toward questions of, 'What does it mean to have this content? What are you charging for? What does it mean when commercials are in, because commercials are how people pay for video? What happens when they can DVR it and skip the commercials? What happens with a show like American Idol?' You can see all the threads starting to come together."
"[J. Allard] was talking about more memory that was going to go in the box. But without debug memory, more memory didn't matter to anybody. If I can't have extra memory on my debug box to test the extra memory, that's not really extra memory. I kinda made that obvious in the meeting. He wanted everybody to be excited about the extra memory, and I was like, 'Thanks, but no thanks.'" — Ric Neil
Until Xenon, game developers had traditionally been granted complete access to every resource in a game console, to what developers call "the metal." Only very rarely would a console maker create software to run between the game and the metal, and so game developers had become used to having access to all of the RAM, all of the computing power and all of everything else inside of a box for running their game.
Xenon would not only be a revolution in how gamers used a console, but also in how developers used it. Live Aware added features that had to be running continuously, between the game and the metal. And a lot of developers didn't like that.
While the Xbox engineers used the Book of Xenon to plan for how the hardware would be built and the software would be coded, the account managers at Xbox's client service department, Advanced Technology Group, were carrying the Book of Xenon to game developers around the world and evangelizing the benefits of the always-connected console. This did not always go well.
One concern in particular loomed large in almost every conversation with potential Xenon developers, and no amount of massaging from ATG managers would make it go away. The problem was Xenon's RAM. There wasn't enough of it. Microsoft had planned to ship the box with 256 MB of shared RAM. Game developers were arguing for twice that. And few argued more loudly than Epic Games' CTO Tim Sweeney.
Sweeney's argument: Xenon's advanced CPU and GPU were great, but without enough memory, games wouldn't be able to take advantage of them. It didn't matter how powerful the box would be, he argued; without enough RAM, games would look terrible.
"Eventually they started making images of what they could do with this amount of memory and what they could do in the same scene with a larger amount of memory."
"There had been meeting report after meeting report," says then Advanced Technology Group Manager Jeff Pobst. (He's now president of Hidden Path Entertainment.) "Every time I did a visit [to a developer studio], I wrote up meeting reports. And there were a ton of them. The constant theme throughout all of them was, 'This looks really good but there's not enough memory.' ... [Developers] knew that it was a hard argument to make from a nontechnical point of view. They were making lots of technical arguments. We were trying to communicate those [to Microsoft]. Eventually they started making images of what they could do with this amount of memory and what they could do in the same scene with a larger amount of memory. That was probably the best way to communicate to execs."
Epic prepared a demo of a scene in its upcoming game Gears of War using only 256 MB of RAM and put that side by side with the same scene using 512 MB. In a meeting with Xbox SVP Robbie Bach, Pobst was asked to explain why developers were arguing for more RAM. As both an engineer and an experienced producer, Pobst presented the arguments and, most compellingly, shared Epic's visuals. The result was dramatic.
The demo finally demonstrated clearly what Microsoft would be sacrificing if it went with the lesser amount of RAM. The two side-by-side scenes were like night and day. On the one side, the 512 side, was an HD video game with enhanced lighting and shaders that looked awe-inspiringly next-gen. On the other side, the 256 side, was a game that could have been seen that very day on any current game console. At a time when customers were eager for new experiences, a Xenon with only 256 MB of RAM would be giving them an old one. Xenon would get its extra RAM.
According to Xbox spokesman Larry "Major Nelson" Hryb, after the decision was made, Microsoft’s CFO called Epic’s Mark Rein to complain that the studio had cost Microsoft $1 billion. Hyrb reported that Rein replied, "No, we just did one billion gamers a favor."
"We felt that we needed a 'voice' for online and committed to bringing some folks in to be a presence with our audience, not exclusively online, but really using online in a different way to talk to people than anyone had in the past." — J. Allard
Before Facebook and before Twitter, there was Xbox Live. In the early 2000s, most social activity online was in discrete communities or forums. Passionate gamers would gather in a kaleidoscope of different places, sharing experiences in conversations that were often difficult to track. Microsoft was concerned that it might not be hearing enough information about the user experience of Xbox Live and that, when the company had something to say, it would be hard to speak directly to gamers.
The Xbox team took inspiration from MTV. Once upon a time, MTV was a cable network that showed only music videos. The hosts, called VJs, served as the voice of MTV and became celebrities in their own right, sharing information directly with viewers and hearing back through fan mail and at events. Xbox Live needed its own VJs. And that's where Larry Hryb and Christa Charter came in.
Charter was a member of the Xbox Live community management team. She was tapped to work alongside incoming radio host and gamer Larry "Major Nelson" Hryb.
"One of the challenges we had in the early days of Live, and that's one of the reasons Ben [Kilgore] hired me, was, he's like, 'Look, I need you to hang out and read the forums and see what's going on,'" says Hryb. "He said, 'Keep an eye on things and let us know if anything seems weird.' In other words, they didn't have a lot of reporting back then. ... The original vision was to be the canary in the coal mine."
That canary in the coal mine vision quickly morphed into Charter and Hryb becoming the most visible faces of a growing team that was attempting to gauge the interests and experience of an exponentially increasing number of Xbox Live users. Charter and Hryb hosted video shows and put themselves in the limelight whenever a change or new feature for Xbox Live was announced and represented the company at events like E3.
The initial result was Inside Xbox, a video show hosted by the duo that would be a clearinghouse for everything happening on the service. Just like the VJs of MTV, Charter and Hryb became the voices of Xbox Live, speaking directly to gamers and hearing back from them in a way that a company like Microsoft never could. The experience was not without some pain.
"At the time I was like 37 years old with two kids," says Charter. "Not exactly what you would think of to put on your TV through your Xbox."
"At the time I was like 37 years old with two kids," says Charter. "Not exactly what you would think of to put on your TV through your Xbox. So I took a bunch of shit for that. ... It was just, you know, death threats. Really specific. What caliber of gun they're going to use. How they're going to break into my house. Threats against my children. ... It was new for Microsoft."
It would ultimately be an experience that Charter, wife of a Marine gunnery sergeant, could no longer abide. She left the limelight in 2010 and worked for a time on other projects at Microsoft before resigning from the company entirely to begin a consultancy business and, eventually, a publishing outfit. These days Charter is a novelist, writing a series of murder mysteries based loosely on her experiences at Microsoft.
Overall, in spite of the threats and negative pressure, Charter has mostly fond memories of her time as a voice of Xbox.
"I really liked that I got to travel a lot," Charter says. "I got to meet really cool people. Some TSA guy in [the] Honolulu airport would say, 'Hey, you're Trixie 360!' My mom and my kids and my fiancé at the time were there. That's something that not many people would get to experience. It's really cool and kinda creepy. One or two will do you for life. I'm really grateful for all the cool stuff I got to do, but I'm also glad it's over. I'm really glad to be behind the wall of the computer monitor again."
Hryb is still at Microsoft, still creating Inside Xbox and still serving as the face and voice of a global multimedia brand. He was recently at the forefront of the Xbox One launch, emceeing events around the world in support of Microsoft's latest console.
"We had these conversations internally where it's like, no one is ever going to thank us for doing our job," Hryb says. "We are the dial tone of console gaming. You pick it up and expect it to be there. [Gamers] get angry when the dial tone doesn't work. ... I've been on call for essentially 14 years."
"When Braid blew up I was over the moon. I really wanted [Xbox 360] to be the first console to fully embrace the independent developers." — J. Allard
In the spring of 2004, Allard and Xbox Product Manager Boyd Multerer were due to talk about Xenon at the annual Game Developers Conference in San Francisco, to begin promoting the box to potential game developers. The only problem was that Microsoft had not yet announced Xenon. For all intents and purposes, the box didn't yet publicly exist, and by the next GDC it would already have shipped. So what could Allard and Multerer say to game developers about Xenon right now, before it was too late — without confirming its existence?
The Game Developers Conference was an opportunity for the key Xbox leaders to speak directly to some of the most influential game makers in the world. The conversation they wanted to have was about the next-gen box. High-definition graphics, bigger processors and much bigger games. A move to HD would mean games would look better, and gamers would expect a significant jump in quality. But the quality increase would come at a cost.
"From a developer’s point of view, your cost didn’t double. It’s going up three or four times."
"The consumers, they all think about how much better the games look," says Multerer, "but from a developer's point of view, your cost didn't double. It's going up three or four times. All those pixels have to get filled by artwork, and that's where all your money goes. [Microsoft was] trying to figure out how to have this conversation about next-gen boxes, about high-definition boxes, to put the stake in the ground that the next generation is about going to high definition, without admitting that we're building the box. So [Microsoft] came up with the name XNA like a week before the conference."
XNA, synonymous with DNA, evoked the idea of building next-gen functionality directly into the core of Microsoft's next box. The GDC presentation, delivered by Allard, Xbox evangelist Chris Donahue and Advanced Technology Group lead Laura Fryer consisted of showing off three small games with Xbox Live-esque features built in. Game site IGN suggested the presentation resembled a graphics chip tech demo, but claimed the presentation hinted that Microsoft was thinking about the next-gen box. Which is exactly what Microsoft intended.
Behind the scenes, XNA was just a placeholder. The name had no real meaning, no definition and no plan. It literally was just a tech demo, as IGN had surmised, but the tech, at that point, was still speculative.
"They used that [presentation] to say, 'Hey, look, this is really a conversation about game developers and what it means to build high-definition games," Multerer says. "It was a good story. It was the right story to tell at GDC, the right conversation to have. But then it was a few months later, like June, and we all had a couple of off-sites, me and J. and a couple of other people. We said, 'OK, let's figure out what XNA actually is now.'"
Multerer began working on XNA full time in August of 2004. His first task: Figure out what XNA would be.
Multerer's team decided to tackle the problems of creating games in a post-HD world head-on and build a platform game developers could use to create games quickly and cheaply, and the team would offer them on a service already tied to a major console. XNA would be a game creation engine, helping to inspire and empower game makers who might not have the resources to create fully HD games. And it would rely on Xbox Live as a means to disperse the XNA development tools to potential developers and also as a marketplace for their games. The XNA plan, if it worked, would become a self-sustaining initiative, benefitting developers, consumers and Xbox. And it wouldn’t be possible without Live.
"[This] was another great big debate," says Henshaw. "Will it work? What are the right price points? How big does the catalog have to be? Are publishers gonna be on board? Is there a different relationship we have with publishers for smaller arcade games that are downloadable? Does it dilute the entertainment value proposition? You can imagine the debates we had. But like everything, usually the debates wind up in ... 'We're Xbox! Let's try it and see what happens!'"
And so they tried it, and what happened, among other things, was Xbox Live Arcade.
"The focus of the Xbox 360 was, 'Let's just make the developer's life way easier than PlayStation does,'" says former ATG Account Manager Ric Neil. "And it worked. It went from everybody developed on the PlayStation 2 and ported to the Xbox — to the whole community then making things on the Xbox 360 and porting to the PS3. People at Sony were telling developers, 'Oh, you could get a lot more out of our box if you developed these custom shaders and did things more custom!' The same argument [Microsoft] was frustrated with on [Xbox Prime], Sony was [now] hearing on PlayStation 3. Developers are like, 'I'm not gonna make any more money by spending $300,000 on a new shader. You're going to get the Xbox 360 port.'"
"For 100 bucks you could build a game and ship it on your [Xenon]."
"I think we built one of the best independent gaming frameworks ever," says Multerer. "You've heard of the Unity engine? That's built on it. It's built on XNA. That technology is pushing games on all these different devices. ... The thrust we took with XNA was, 'Let's go build an independent game developer program.' And we did. ... For 100 bucks you could build a game and ship it on your [Xenon]. ... Super proud of that program overall."
Multerer believes the current wave of indie game development had its roots in Microsoft's XNA. He goes so far as to take direct credit for helping the indie explosion. Whether or not that's justified, XNA introduced the first widespread indie game development program on a major game console and helped bring about games like Fez, Dust: An Elysian Tale, Bastion, Terraria and Serious Sam Double D.
"I think the first people who went into college and took comp sci because of XNA have now graduated," Multerer says. "A bunch of them have taken jobs at Microsoft, and they've been hunting me down over the last two months to say thank you. I've had this stream of people I don't know coming to talk to me about XNA. It's been really kinda neat."
"On launch day, or night, at the local Best Buy in Bellevue, WA, there was this big launch event. Bill Gates was there. All the media's there. There was a line of probably 500 customers waiting for the midnight madness thing. They shipped the [Xbox] team down on some buses. Everybody's partying. It's this super festive thing. It's a cold November night. We go down there, and I was sitting next to a guy, a peer of mine. ... We had one of those relationships where we'd butt heads once in a while. We were both strong-willed people. You walk off that bus and there's 500 people waiting in line at midnight chanting, 'XBOX!' It's your little rockstar moment. You walk off, and I looked at this guy, and I shook his hand, and I was like, 'This is why we do what we do.'" — Ben Kilgore
"You walk off that bus and there’s 500 people waiting in line at midnight chanting, ‘XBOX!’ It’s your little rockstar moment."
Throughout 2005, final details of the Xenon launch were released (or leaked) piecemeal from Microsoft. The box would be called Xbox 360. It would ship in two flavors: one with a hard drive and one without. And it would be always connected to Xbox Live.
"This whole idea of Microsoft building a game console and operating a service, this was still brand new back then," says Whitten. "It was certainly still in a time where the company was trying to figure out what had happened. 'Was this an interesting experiment? Was this a really deep piece of value? Did we really understand what we were going after?'"
Microsoft managed to convince a number of game developers and publishers to build launch titles for the Xbox 360, games that would inspire gamers to embrace the new device, including Madden NFL 06, NBA 2K5 and Need for Speed: Most Wanted. And Microsoft had its own titles, including the highly anticipated shooter sequel Perfect Dark Zero. But the big gun, in a number of ways, was Call of Duty 2.
"Here at the office, we have the first disc off the press from Call of Duty 2," says Infinity Ward Executive Producer Mark Rubin. "Because we were the first disc in manufacturing. That actually is the very first 360 disc ever made. It's in a plaque. Microsoft had sent it to us a while back, as a sort of a gift."
The first Call of Duty had been a moderate success on PCs, but its sequel was the first in the series developed primarily for a console. No one knew what to expect.
"The feedback for Call of Duty 2 was amazing," says Rubin. "People were so excited. They loved the game. We had one of the highest attach rates ever for a video game. ... No offense to the Perfect Dark Zero guys, but that game kind of flopped by comparison. It was Call of Duty 2 that really owned that launch for 360."
The game launched a new wave of monster hit games on consoles, driven by competitive multiplayer shooting, and gave rise to the multi-billion dollar industry that is the Call of Duty franchise. But the game itself faced a number of challenges. Not the least of which was the fact that the final hardware for the Xbox 360, in part due to the decision to add more RAM, was very late getting into production. It would be late into 2005 before the Infinity Ward team even saw the final device.
"Toward the end, when we were expecting to get final hardware way earlier than we got it, it starts to terrify you a little bit."
"There were milestones missed," says then Infinity Ward CEO Vince Zampella. (He is now CEO of Titanfall developer Respawn Entertainment.) "That puts us in a precarious position. OK, now our schedule just slipped, because we can't develop things while Microsoft is running late. ... Toward the end, when we were expecting to get final hardware way earlier than we got it, it starts to terrify you a little bit. 'Hey, wait a minute, are we gonna make this? Is the hardware going to be ready?'"
It wasn't an idle question. Because of the 360 hardware delays, Infinity Ward was beginning to seriously doubt that it would be able to finish its game on time to ship alongside the new console. If it couldn't, it would have to wait — perhaps as long as an additional year.
"If the hardware slipped past Christmas — let's say they slipped it three months, or even one month — we can't finish the game," Zampella says. "And even if we could, what would you do with it? ... Being a launch title has its own set of problems in that you're building to hardware that doesn't exist yet. We didn't get final hardware until two months before we certed."
In the end, Infinity Ward managed to finish, and Call of Duty 2 shipped on time as a 360 launch title. It sold a quarter of a million units its first weekend, eventually going on to sell over two million copies in all. Modest for today's standards, but impressive at the time for a third-party shooter on a gaming console.
Xenon’s "Live-enabled" console concept very quickly proved its worth. It launched a year before Sony's PlayStation 3 for $100 less and with more launch titles. Sony, on the other hand, was widely considered to have flubbed its PS3 launch. Delaying its box to accommodate its proprietary Blu-ray technology cost the console maker a year head start and increased the price relative to the Xbox 360. As a result, many gamers, hungry for a next-gen experience, decided not to wait for PS3 and bought an Xbox 360.
The Xbox team used the lessons learned from its war game to conduct a victorious, if not perfect, launch. The final hardware came late, and not (as it turned out) without some production problems and disappointing software, but on the Live side everything worked as designed. Gamers were finally getting their next-gen game console, with Live built right in. And they were getting it a year before Sony’s PlayStation 3.
Meanwhile Sony made almost every mistake imaginable, and it cost them. Over the next eight years, Xbox 360 would leverage its early lead into a decisive victory, outselling the PS3 by four million units worldwide and 10 million — almost two to one — in the U.S., making Xbox 360 — and Xbox Live — the gold standard for online console gaming.
For Xbox Live, however, the story was not over. Launching as the heartbeat of the Xbox 360 meant more features in the hands of consumers, and more changes for the infrastructure running the service. But it also meant big changes for Team Xbox itself.
Lurking on the horizon were major changes in Xbox leadership — and a game that would turn console shooting into a global phenomenon.
"It felt great, to be honest. I mean, in a sort of sadistic way, it was amazing. ... When Microsoft, one of the biggest companies in the world, is struggling to handle the amount of people that want to play your game online, it's a really strange experience. A good experience, but a strange experience." — Mark Rubin
"When Microsoft, one of the biggest companies in the world, is struggling to handle the amount of people that want to play your game online, it’s a really strange experience."
Two years after the launch of Xbox 360, and one year after the launch of Sony's PlayStation 3, it was becoming clear that Microsoft had done it. The console war might not have been over, but the winner was clear.
By the holiday season of 2007, Xbox 360 was outselling the PlayStation 3 by almost two to one. Halo 3, the sequel to the game that had proven the Xbox Live service in 2004, had launched in September to record-breaking $170 million first-day sales, a new world record. Six weeks later, the next most-anticipated Xbox 360 game of 2007, Call of Duty 4: Modern Warfare, took its bow, coming in as the third highest-selling game of the year, with Nintendo's Wii Play between them. By the Christmas holiday, so many people were playing both shooters that a failure of Xbox Live seemed all but inevitable.
"It was just the volume was so high," says Zampella. "They weren't anticipating it. It's the perfect storm of, 'people want this game so much; there's so many people playing it that the system just can't handle it.' That's a good problem to have."
Jon Shiring, then the network programmer at Infinity Ward (now at Respawn), was visiting relatives when he got the call.
"I remember, at the time, there were gaming news stories talking about how you couldn't play Halo on Christmas Day," Shiring says. "I was thinking to myself, 'Well, it's not Halo's fault!'"
Call of Duty had experienced an odd transition with its relationship to Microsoft. Whereas the Xbox 360's other hit game, Halo 3, had come from a team owned by Microsoft Game Studios, Call of Duty was a third party game, and — at first — wasn't getting the same amount of attention from account managers as Halo.
"At the launch of the 360, they were much more excited about Perfect Dark Zero and those games," says Shiring. "Call of Duty 2 was kind of under their radar. ... At the beginning they had these high-level astronaut engineer views of, 'This is what would make people happy.' Eventually they saw what we were doing and kind of changed that [perspective] entirely. ... But it was a journey to get there."
The start of that journey was the Call of Duty 4 beta. The beta was another rare event for a game console, but it was an event that, like so many of Xbox Live's innovations, would become relatively commonplace. A small number of players were invited to help playtest the Call of Duty 4 multiplayer before the game was launched. Not only was it a chance to find problems before the game shipped later in the year, but it was also a chance to show just how popular Call of Duty multiplayer could become. While the beta proved, overall, to be a major success, it inadvertently triggered another first for Xbox Live: patching a game while it was running live.
Right after the beta program was certified by Microsoft, Shiring found a bug in the code that would strand players on the connection screen if any bit of data got lost on the way through the internet. Normally, networking will allow for packet loss and send a request for a repeat of the data or else try to get by without it. But this bug would freeze a player's game if it lost a packet, which would become a showstopper for a large number of players.
"We couldn't patch the game, because they just basically told us, 'There's no way to get a patch through in the time frame from your beta,'" Shiring says. "So we ended up doing some crazy server hacks."
Shiring convinced Microsoft to let him fly up to Redmond with the game build on his laptop and plug it directly into an Xbox Live developer kit to update the servers. He was, in effect, altering the game's server code while the game was hosting thousands of players. It was not unlike repairing a plane's engine while the plane is in flight.
"That's kind of a nightmare for an engineer," Shiring says. He tried to solve the problem on the server end by programming the servers to send a flood of packets in response to a user request, effectively shotgunning packets at the player's Xbox, hoping that at least one of them would get through. The hack solved the issue for most players, but it wasn't perfect. Players on especially laggy networks were hit hardest. But most players never knew there was a problem. They played the beta, told their friends and word of mouth grew.
By the time Call of Duty 4 was due to launch, both Infinity Ward and Microsoft had developed an inkling of what was in store, but nothing they had experienced prepared them for the flood of players that actually arrived.
"We weren't prepared for the concurrency, the number of players that were coming in between Call of Duty and Halo 3," says then ATG Account Manager Sam Charchian. (He's now an account manager at Sony.) "It was just an explosion that holiday, with those two titles hitting together at the same time over the holiday. ... One thing went down and it started taking down other pieces."
The weekend of the Christmas holiday, immediately after the launch of the two biggest-selling Xbox 360 games of that year, Xbox Live went down. Gamers attempting to log in to the service to play their games were greeted with an error message.
The exact source of the crash was a piece of code in Call of Duty 4. It was a small subsystem that would tally the number of players live in the game. When you signed on to play, you could see how many other players were playing. It was a neat but unnecessary flourish, and it ended up breaking the game — and Xbox Live.
As new players logged on, the first thing their game would attempt to do was page that system, to register with the counter tallying the number of players. If the counter system didn't respond, they would keep trying. Again, and again, and again. When the counter went down on Christmas weekend, the result was an inadvertent DDoS attack, first on Activision's servers and then on Xbox Live. Worse, it wasn't just Xbox 360 devices that were affected.
By 2007, Xbox Prime and Xbox 360 were not the only devices using Microsoft's online service. The fledgling Zune music player had also tapped into the service. The Call of Duty 4 crash didn't just interrupt Call of Duty and Halo games; it also disrupted service to everything else Microsoft had tied into Xbox Live. Which turned out to be everything.
"The problem was, when we took down Xbox Live, the way Microsoft was set up, we took down everything," says Rubin. "Hotmail didn't work. Anything Microsoft Marketplace didn't work. The whole Zune thing didn't work. ... Nothing Microsoft worked. You couldn't even get to the Xbox website. So yeah, I very much remember ... being on the phone at 3 in the morning with them, trying to help them get back up online and work with them on getting that done. To their credit, that was a very difficult lesson for them that not only affected, obviously, the Xbox division of Microsoft, but every other division. And so massive steps were taken."
While Call of Duty 4 was busy breaking Xbox Live, plans were already underway at Microsoft for the next version of Xbox Live, and the next version of Team Xbox itself. Management shuffles would soon introduce a wave of changes, turning the experimental playground at Millennium E into something that would look more familiar to the rest of Microsoft. The new leadership's first priority would be stabilizing Xbox Live. Their first step: They called the fireman.
"There's been days at Xbox where it's felt similar to those days back in the fire department. It's definitely a higher-stress job when things aren't going as well as you hope." — Derek Ingalls
Derek Ingalls is a quiet, reserved individual. He carries himself like a gunfighter, looking more hard laborer than hardware engineer. He has the hands of a man who's used them and the eyes of someone more used to looking for trouble around him than inside of a computer. When he walks into the room and sits down, you say to yourself, "This is a man who's seen things and survived." And that's no coincidence. Before he came to work at Microsoft in 1997, Ingalls was a fireman.
"I’m like, ‘Nobody’s going to die today. I know what that looks like. Calm down.’"
"When I was running Exchange internally," says Ingalls, "if Bill [Gate]'s Exchange server was down, there were people running up and down the halls like the building was on fire. I'm like, 'Nobody's going to die today. I know what that looks like. Calm down.'"
One of the hardest lessons the Xbox team learned in its first five years was the value of what's called "change management," the process of keeping track of changes. It sounds simple, but with such a large and growing service that impacted multiple moving parts at Microsoft and elsewhere, the Xbox team often found itself playing catch-up with its own activity and fighting problems it had caused itself.
When Microsoft needed someone to step into Xbox and start running it like a business in 2008, it tapped Ingalls, an expert in change management who was, at the time, running Exchange. Ingalls likens the job to making sure the toilets flush.
"As long as they flush, everybody’s happy, but nobody’s going to pat you on the back for it."
"As long as they flush, everybody's happy," Ingalls says, "but nobody's going to pat you on the back for it. But in the event that they don't, people make a lot of noise about it."
Ingalls's first week on the job as Xbox Live's operations general manager was right before the launch of Grand Theft Auto 4.
"I don't know if you remember," says Neustadter, "but the CEOs of Rockstar made a public comment, something along the lines of, 'We'll be so popular we're going to burn Xbox Live down.' So we were very determined that that wasn't going to happen."
With the chaos of Christmas the year before still fresh in its mind, the Operations team was talking about adding server capacity to accommodate GTA4, while the facilities team was talking about problems it was already having with overheating in the server farms. Ingalls, present for both conversations, realized that not only did Xbox Live have a problem, nobody else knew it.
"I'm watching this and trying to put the two conversations together. ... And I'm doing the math," Ingalls says. "'Wait a minute, if we're having power and space and cooling problems, and there's this big game coming, and the numbers you're talking about ... are we in trouble?'"
Ingalls was an unlikely Microsoft engineer. His twin brother Dustin already worked at the company while Ingalls was with the fire department. The two would have lunch and Ingalls would tell his brother about surviving a bad fire or watching a child die, and Dustin would tell him he needed to find another job. Eventually Ingalls took a break. Redmond FD has a long-established sabbatical policy. Firefighters are encouraged to take a two-year leave, after which they can return to their exact same job at their exact same salary, with no loss of benefits or loss of seniority. Most who take it come back. Ingalls took it and went to Microsoft.
Ingalls believes his fireman training prepared him well for running critical services at Microsoft, especially in knowing how to prepare for accidents — and prevent them.
"It's just unacceptable to ever be surprised," says Ingalls. "It's great for the business to say, 'Hey, wow, business is booming and we didn't realize how great it was going to be!' But when you get down to it, we know how many consoles are going out the door. We know how many Live subscriptions we're selling. We know how many game discs we're printing. We know the code that we wrote. We know how it runs in test and in stress. I think there's no excuse, ever, to be surprised."
Overseeing Xbox Live Operations, Ingalls focused on prevention, building up processes and training XOC staff to handle crises efficiently and — most importantly — consistently. And he hired a team of engineers to work separately from Xbox Live Operations to ensure that the training held — and remained fresh.
The one problem with prevention is that it can breed inattention. If a fire never starts, for example, the people who put out fires might forget how. That's why firefighters train by setting fire to abandoned houses. For Xbox Live, Ingalls needed a way to burn down the house without waiting for someone to do it for him. So he started doing it himself.
Ingalls oversees a team at Xbox called "Team Chaos." Their job: Inject faults into Xbox Live so that the XOC team can find and repair them. Ingalls believes this keeps the XOC team in shape and ready for when something happens that it didn't cause itself.
"We just don't tell anybody, and we let 'em go," Ingalls says. "We grade ourselves on how quickly we identify and shut down what's going on."
During his first week on the Xbox team, while putting the pieces together regarding the impending doom of the GTA4 launch, Ingalls also quickly realized there was no written process for anything at Xbox Live. For any given event, Operations Center employees would line up behind Neustadter, a solution would pour out of his brain and then people would do what he told them. It worked, to a point, but for a large and growing service like Xbox Live to have its entire operation residing in one person was potentially catastrophic.
Ingalls moved quickly, established procedures, separated existing processes from their habitat in Neustadter's brain and began a methodical, but rapid, process of splitting Xbox Live into multiple server farms with its now legendary geographical redundancy. The result: a dramatic increase in uptime and a reduction in overall outages. Xbox Live would now be ready for whatever Microsoft had in store for it. Perhaps surprisingly, the person most receptive to the changes was Neustadter himself.
"[Ingalls' arrival] was great for me, because Derek is great at all the things I don't like to do," Neustadter says. "Deep down, I am a technical person, and I'm happiest on a whiteboard designing stuff and trying to solve cool technical problems to let games do new stuff. ... That's what really jazzes me about the job."
In 2008, the Operations team was around 50 people, and managing these people was Neustadter's full-time job. No time for doing technical work, no time to go hands-on with games. It was sucking the enjoyment out of his day and he was happy to let go of running the XOC and get back to working with the creators on making games. Since Ingalls' arrival, Neustadter has spent the majority of his time working with the teams behind Halo, Call of Duty, Gears of War and others to make sure their games work as well as they can on Xbox Live and that Live has the tools it needs to support them. He's gone back to being a game engineer, and he couldn't be happier.
"That's something I hadn't gotten to do a lot of since probably Halo 2," Neustadter says, "because I was busy doing these other things. ... That's part of the reason why I'm still here after all this time. I get to work on really cool problems every day. ... I spend my day sitting here solving problems for Call of Duty, or right now Titanfall, Sunset Overdrive, Destiny, whatever's coming next. Then I get to go home and play them as a customer."
For Neustadter, the evolution of the Xbox Live team just makes sense.
"We went from being a startup to being a big business," he says. "Used to be we could kind of get away with doing whatever the heck we wanted, because we were this little skunkworks project. We now get mentioned during SEC quarterly conference calls about the performance of our business. ... We have had to grow up."
In May of 2008, when Grand Theft Auto 4 launched into the world, it broke all standing records for first-day game sales, moving more than 3.6 million copies in its first 24 hours, selling over a million more copies its first day than Call of Duty 2 sold in its entire lifetime.
Xbox Live did not crash.
Time to penis
"Hacking on Call of Duty started to get totally out of control around , with people hacking the game and custom lobbies and all that kind of stuff," says ATG Account Manager Sam Charchian. "We built this really tight integration between [Infinity Ward's] anti-hack team and ours inside, because Microsoft has their own, what they call the Xbox Live Enforcement team. We had their anti-hack/enforcement team come spend a day with ours and talk about how they could work together to hopefully defeat these people.
"One of the funny things that came out of it ... we're meeting with the Xbox Live enforcement team, and [there was] a woman on the team who's in her early 20s. She was explaining that she spends a good part of her week going into Uno games with the camera on and basically just waiting for dudes who will see a woman playing Uno with her camera on and expose themselves.
"I was like ... 'Really? How often do you actually have dudes, you know, whip it out?' And she said, 'Eh, my time to penis is about 14 minutes on the average.'
I mean, she would sit on there and just ban these guys when she ran into them, but apparently it was a serious problem. Part of what makes it extra sad is that it's Uno, which you think would be a family-friendly game for people to play."
"A lot of folks don't like change. That's when the people who wouldn't normally be talking come out and say, 'Oh, I hate this.' They just don't understand what's going on. When you look back, it makes perfect sense, but at that moment, not everybody really gets it." — Larry Hryb
The sweeping changes to Xbox and Xbox Live were not only promoted by the growth of the service, but also by movement at some of the highest levels at Microsoft. By the end of 2007, Allard, the visionary leader whose dream of a "Snow Crash box" to unite the world had spawned Xbox and Xbox Live in the first place, had moved on to oversee the production of Microsoft's next big thing: the ill-fated music player Zune.
Microsoft's Marc Whitten had stepped in to replace Allard, with Allard's protege (and sometime critic), Boyd Multerer, filling in on the idea side and Executive Producer Ben Kilgore managing production. Together the three men would oversee the next phase of Xbox Live and eventually move the entire Xbox team away from its offsite headquarters at Millennium E back to the Microsoft mothership in Redmond.
"The fun period for Xbox was definitely over at that point. That’s what everyone thought."
"The fun period for Xbox was definitely over at that point," says Ingalls. "That's what everyone thought. For the first couple weeks, there was an amazing amount of trepidation. ... Teams get really passionate about the work that they do and the people they work with. All the big leaders have a posse, and so everybody's always like, 'What is that going to mean? What is that like?'"
Xbox had, at one time, been an experiment. No one, not even the Xbox team itself, knew for sure that it would succeed. The structure required the team to create something completely new in a completely different category because Microsoft was a culture that could encourage people to break barriers and not be afraid to fall on their faces. In other words, a culture completely unlike the entrenched, politics-heavy corporate culture at Microsoft. Now that Xbox had become a success, it was time to rein that counterculture in.
"I had an awesome time at Microsoft," says Charter, "but as the Xbox division got bigger and more successful, it became far more buttoned up and corporate than it was when we were over on the redheaded stepchild campus all by ourselves, just spending Bill's money and doing stuff that everyone would live to regret. The more corporate it became, the more it was like, 'I don't really fit in here.'"
Externally, the Xbox team was pushing toward what would be the first major update to Xbox Live since the launch of the Xbox 360. Dubbed "The New Xbox Experience," or NXE, the update would dramatically alter almost every aspect of how users interfaced with the service — and how they would use the console itself. It would be a major test of both Whitten's new priorities and the concept of a long-lived console.
At the time, few people outside of Microsoft knew just how long the Xbox 360 was intended to last. Game consoles typically had a four- or five-year lifespan. In 2008, Xbox 360 was in its third year of life, at which point most observers would have expected to begin hearing about a new console. What they heard instead was about an an update to Xbox Live.
"There was this process internally at Microsoft called strat review," says Whitten, "where each year, every part of the company would come in and pitch their strategy. As you can imagine, it's lots of PowerPoints and lots of Excel spreadsheets and all this stuff. I had been thinking a lot about this idea of, 'Well, what does it mean to really open up Xbox Live and to make it more visual, to help people discover content faster, to add new functionality?' And so instead of a PowerPoint and an Excel spreadsheet, I had the team build a video, a sort of two-minute video of what Xbox Live could look like. I also wrote the review, a magazine review article. ... I would show it to people and they'd say, 'Yeah, but it's not a new console.' And I'd say, 'Exactly.'"
The core of the NXE was a change in what's called the UI, or user interface. The original Xbox Live used a UI called Blades. Each type of activity on Xbox Live had its own blade, and users switched between blades using buttons on the controller.
"At the time we thought [Blades] was the most amazing UI in the world," says Kilgore. "We used to just sit there going back and forth between the blades and high-fiving each other. ... It's crazy what you do at 2 a.m. after working 100 straight hours. Sometimes you might lose a little perspective."
As cool as Blades may have been, Kilgore and others at Microsoft worried that the UI didn't showcase all that Live could do. And the more Live was able to do, and the more users were able to do with it, the less powerful Blades proved to be.
"You didn't have this sense [with Blades] that you were part of something bigger," Kilgore says. "What we wanted to do with NXE was to go and create the sense that you're a part of something bigger. Show you what people are doing. Show you all the different ways that you could be online and participating. ... We want to be Disneyland, not Coney Island, [and] NXE was the map of the park."
Microsoft had been sending a survey to users logging out of Xbox Live. One question: Why are you leaving? The answer, more often than not, was: "There's nothing to do."
"Then we'd say, 'What if you could do this?'" Whitten says. "They'd say, 'Well, that would be cool.' And of course they could do that on Xbox Live; they just didn't know it."
The NXE would change all of that. It would introduce a more streamlined interface, similar to the Windows OS of the time. It would bring some of Live's deeper functions, like the marketplace and messaging, to the forefront, showing users a hint of everything Live had to offer whenever they turned on their Xbox.
Most importantly, NXE would help Microsoft realize the possibilities of a console that could reinvent itself using the exact same hardware — and extend the lifecycle of the Xbox 360 for another unprecedented seven years. And it would introduce a video streaming service that would revolutionize the way most gamers used their Xbox consoles forever.
Avatars with guns
The New Xbox Experience update to Xbox Live added a number of features designed to increase the social and casual use of Xbox Live. One of these was the introduction of Mii-like avatars. Users could create an avatar and make it look like themselves and dress it up. But Microsoft, leaning on its philosophy of maintaining a family-friendly, trusted third-party environment on Xbox Live, had relatively strict rules about what could and couldn't be put on an avatar.
ATG Account Manager Sam Charchian recalls discussions with Call of Duty developer Infinity Ward over introducing Call of Duty-related items to the NXE avatars.
"With the NXE dash, we were interested in having our developers support avatars," Charchian says, "whether it meant putting avatars in the game or just creating items that could be sold or used on avatars. Shirts, accessories, hats, whatever. Obviously the Call of Duty guys weren't super interested in having avatars in their game, but they were actually really interested in having props and shirts and hats and all that stuff as things that people could either earn or buy or whatever and outfit their avatar with.
"One of the ones that Jason [West] in particular really wanted to do is, he wanted to have the Call of Duty guns, the full arsenal, available to avatars. So that people could buy whatever their favorite gun is in the game and give it to their avatar as a prop. And we had this really strict anti-gun ... no, you can't have guns. That was the rule.
"We went back on forth on this quite a bit, because he's like, 'No, no, our gamers will really love this. I think you're making a big mistake. It's your biggest game. It's something that our customers would want.' And so I ended up having this phone call with Jason, Vince [Zampella] and probably Mark Rubin and our person who held the policy for avatars about this very topic.
"Jason was really funny. He came well prepared. [The avatar policy manager] said, 'Well, we have a strong no guns kind of policy.' [West] said, 'Well, I notice that you can get the energy sword from Halo.' She says, 'Well, that's not a gun. That's a weapon.' He says, 'Well, that's also very deadly. How is that different?' She says, 'It doesn't shoot.'
"He says, 'OK fine, here's what I'll do. I'll make a gun, but it's not a real gun. It doesn't shoot. It's a toy gun. It looks like one of our guy's. She says, 'No, no, that would still look like a gun.'
"He says, 'Look, here's what I'll do. I'll have it so it's one of our guns, and the avatar, he's gonna point it straight at the camera and pull the trigger, and nothing's going to happen. He's going to get all frustrated. He's going to try to clear the jam. He's going to point it back at the camera and it still won't fire. It'll never fire! He'll just act frustrated and try and figure out what's going on with it.'
"[West is] trying to find any way he could to wiggle around this policy. But our policy person was not having it at all. We ended up just telling them, no, they couldn't do it. I had to mute [the phone] because I was laughing so hard watching Jason work around in circles trying to get this person to see his side. It just wasn't going to happen."
"We knew [in 2005] that the video store would probably be the next thing that we did. ... I'm sure J. knew, when he was doing the first Xbox planning [in 1999]." — Ben Kilgore
Microsoft launched the Xbox Video store in 2006, accessible over Xbox Live from Xbox 360 and Zune devices. It was a rudimentary affair. You bought a movie, downloaded it and, some time later, watched it. Xbox Live would measure your download speed, calculate how long it would take for you to finish downloading the movie versus how long it would take to watch the film, and then let you start watching it when those two numbers became "less than or equal." Simple. State of the art, at the time, but not very exciting.
By 2008 Microsoft and other companies were working on proprietary multi-bitrate streaming technologies that would allow transmission of coherent data a trickle at a time. Multi-bitrate streaming would allow Xbox Live to manage transmission speeds and adjust its own speed to suit whatever was happening in your box, on your internet connection or on its servers. And it allowed for a wider range of activities online than anyone (except perhaps Allard) could have imagined. Which, for most people meant simply: You can press play and watch a movie right now.
At the time, most of the innovation in streaming technology was being driven by video purveyors like YouTube and a small video-rental-by-mail service based out of California called Netflix.
"At the time, Netflix was really early in its transition from its disc-based model to its streaming service," says Whitten. "We had been looking at a lot of the data about bandwidth and the trends around bandwidth, and we saw this huge opportunity, what they brought. They had such a great brand. And so we just started talking together as teams and saying, 'Hey, let's tell you guys a little bit about Xbox Live,' have them tell us a bit about Netflix, and we all saw that opportunity of, well ... as I always like to joke, it's the strategic insight of, 'People really like to watch video on their TVs.'"
In spite of the strategic insight into people's preferred usage for their televisions, there was some concern about what effect streaming video might have on people's preferred usage of a gaming console. Specifically, would allowing such a huge video-related service onto Xbox Live mean taking focus away from the gaming core?
"At the time we just didn’t know. It was this new space. Streaming was still a very new thing."
"At the time we just didn't know," says Neustadter. "It was this new space. Streaming was still a very new thing."
In spite of the misgivings, Team Xbox did what it always had: It took the risk.
Having learned its lesson from Christmas 2007, the operations team approached the introduction of Netflix to Xbox Live as it would any new major game release and crafted a launch strategy to account for a rapid influx of potentially millions of new users — and the resulting net traffic.
"We found a lot of our systems ... we were outgrowing them, and outgrowing them so fast," says Neustadter. "[In] 2008, we had a million users on the service simultaneously for the first time. ... That was really rapid growth. We rebuilt a lot of things along the way in order to make that possible, things an end user might not see. They see Netflix and NXE, things like that. They don't see us swapping out everything under the hood."
While the operations team was swapping out server technology, it was also in deep discussions with Netflix about how the two services, Netflix and Xbox Live, would come together. Xbox Live engineers wanted to understand how Netflix worked, how its queue was managed, what servers were used for what actions and, perhaps most importantly, when a Netflix customer called for support, whose phone would ring: Microsoft's or Netflix's.
For the user at home, watching a movie on Netflix on an Xbox 360 might not seem much different from watching the same movie downloaded from Xbox Video, but behind the scenes it involved the cooperation of two completely separate companies working largely in two vastly different industries. To date, Xbox Live had tapped a dizzying array of talents and expertise from within Microsoft, but almost everything Xbox Live had accomplished had been built using what were, for Microsoft, off-the-shelf solutions and knowledge. Microsoft was an enterprise data and software company, and Xbox Live, while used for gaming, was essentially an enterprise data service. Netflix added a new element and a different type of relationship. Xbox Live was now in the business of Hollywood. And Netflix would be its first co-distributor.
Netflix rolled out to Xbox Live subscribers along with the NXE in November 2008. According to Neustadter the launch was, for all intents and purposes, flawless. The only major problem was the Netflix registration process itself. Users were required to start Netflix on the Xbox, then walk away from the TV and visit the Netflix website, log in, register and receive a code, which they then had to carry over to the Xbox 360 and input using the controller. The process, while tedious, generally worked.
The problem was that, again, no one expected how popular the service would be. The system generating the codes didn't have enough digits. After a certain number of users signed up, the system no longer worked.
"It was running through all the possible combinations too fast," Neustadter says. "We had to actually do an emergency update to the Netflix app to put extra digits in the code."
After the update, no problems. And in the end, the availability of streaming video didn't decrease interest in playing games on Xbox Live. If anything, it increased it. Even today there are more Xbox 360 units being sold than either Microsoft or Sony's latest-gen offerings. And concurrent users for both online games and video streaming have continued to steadily increase, year after year.
"When we crushed the vision for Xbox 360 down into a single sentence to inform the brand work, etc. ... we got to 'A living entertainment experience powered by human energy.' This was in 2003. Now fast forward through everything you know since 2003. ... I bet many people don't even remember that verbatim, but they live it everyday. We built a culture around purpose and principles that could live on, independent of leadership changes and specific words. That's probably what I'm most proud of in terms of my contribution to the team." — J. Allard
The NXE launched almost five years to the day from the launch of Xbox Live itself. It represented more than an Xbox Live UI update. It was nothing short of a reboot of the entire service itself, the third generation of a technology that was now the driving force of a physical game console.
The geniuses at Millennium E accomplished the previously unimaginable again: the reinvention of a consumer gaming console from the inside out. Xbox 360 users were now using it in ways they couldn't have envisioned when it first arrived in 2005. Allard's dream of a connected machine had finally come to fruition, after almost 10 years, two hardware releases and millions of dollars and man-hours.
"When you think about what Xbox Live was when the 360 launched ... it was a dashboard."
"When you think about what Xbox Live was when the 360 launched ... it was a dashboard," says Activision’s VP of Production for Call of Duty, Daniel Suarez. "It was a dashboard that had some interesting things to it. You had your avatar and some basic connectivity. But look at where it is today versus where it started. Everything from it growing from a social network ... to not just the social network, but the competitive component to that. The idea of it being a media hub. It's a retail store. It's everything from movies, TV, music, pop culture. ... It's become the destination for anybody that consumes media. And it started off as ... for most of us, it was our game machine."
Ben Kilgore credits much of what Xbox Live and 360 became to Allard's original vision, including the sometimes goofy practices and beliefs that started from Allard's reading of the novel Snow Crash and resulted in a billion-dollar business.
"We sat down with the engineering teams and the marketing teams and some of the game teams, and we talked about, ‘What’s the true North Star for 360?’"
"We had an exercise J. made me do," Kilgore says. "Drove me crazy at the time, but it was the right exercise. ... It was called the North Star, creating the North Star for 360. It was basically an exercise where we sat down with the engineering teams and the marketing teams and some of the game teams, and we talked about, 'What's the true North Star for 360?' ... We used that to drive a lot of the final features. As we tuned things in the last six months of the project, that's what we went back to. It was our North Star. 'Should we do this or do that? Let's look at the North Star and see, have that help guide us.' That was designed to be broader than just core gaming."
For Larry Hryb, who was hired at the start of development on "Xenon" and who remains a key member of Team Xbox to this day, the evolution of Xbox Live represents nothing less than the evolution of Microsoft itself.
"To really see Xbox 360 and Xbox Live become one — they were inseparable," Hryb says. "In order to make Xbox 360 light up, you had to be connected. That's where your friends were. I remember all of these moments when we updated the system and new games would ship and we were struggling to keep things going. I love all these moments. I love the startup mentality of those days, experimenting in running the service. Xbox Live was one of the very first early services where people were paying us for our services at Microsoft. There were different levels of expectations. Our goal was to deliver those expectations. I think we set the tone for paid services across the entire company."
For Eric Neustadter, who literally built the service from scratch out of server parts and Cisco routers, witnessing the birth and maturation of Xbox Live has been one of the most extraordinary experiences of his life.
"This has been the professional highlight of my career," Neustadter says, "and I don't expect to ever top it. Which is why I'm still here and why I don't intend to go anywhere. How many chances do you get to work on something this cool from the very beginning? I figure the answer is, if you're lucky, once."
Today, the story of Xbox Live continues to diverge from its gaming origins. Although the service is inextricably linked to MIcrosoft’s game consoles, it now also powers applications across all of Microsoft’s offerings, including the Siri-esque Cortana app for Windows phones. Today, more than 50 million people use Xbox Live, and the service continues to evolve.