Meet the people who keep some of your favorite games running

In many ways, 2013 was the year of the spectacular network outage.

The inability to keep gamers connected online hampered everything from the launch of major video games like SimCity, GTA Online and Battlefield 4, to console services like the PlayStation Network, Xbox Live and, just this week, Nintendo's online network.

Server issues were even among the problems that thwarted a successful rollout of HealthCare.gov.

As Electronic Arts' recently promoted CEO, Andrew Wilson, told Polygon in a November interview, all of mankind is transitioning to a digital world, and that transition brings with it some major issues.

"As all of mankind right now transitions to a digital-type world and a digital-type experience, and you think about the scale and the load that comes from all of mankind transitioning and getting access to information through digital channels, I think what we learned from SimCity was you need a very robust test system — it's very different to the one we used before," Wilson told Polygon. "You need a very robust launch system — very different from anything we've ever had before. You need a very robust follow-up process that ensures that you can remedy issues very, very quickly, and certainly that was on a much more escalated level than we've ever had before."

In many ways, 2013 was the year of the spectacular network outage

As online games continued to crash at launch, and networks stumbled, the more interesting news of the year started to be more about which major online games didn't falter on day one. Somehow, at the top of that list was one of the year's most purchased games: Call of Duty: Ghosts.

Despite the huge numbers and heavy reliance on quick, seamless online play, Activision seems to have a history of shipping Call of Duty titles without the same level of online bumps that other similar games face.

It turns out, that's largely due to two things: an acquisition Activision made more than six years ago as online console games began to come into their own, and the near-complete failure of the Call of Duty franchise's first step toward gaming ubiquity.

DemonWare

In 2007, Activision purchased a small Dublin-based middleware company that specialized in the emerging art of getting both console and PC games online. At the time, Activision's then-CEO, Mike Griffith, said that the company expected online gaming to grow significantly in the next two to four years and that DemonWare would be the group that would help them "eliminate many of the challenges associated with online multiplayer game development."

The idea was that DemonWare would work with Activision's in-house engineers, the same sort of teams most other developers and publishers have, to help smooth out online launches.

DemonWare started off as an independent company in 2003.

"In 2005, when we were still a pretty small group of people, 12 to 14," John Kirk, chief technology officer at DemonWare, told Polygon in a recent interview, "I moved to Vancouver, Canada, to set up a presence. Most of our customers were up and down the West Coast and it was much easier to support them on their own time zones."

At the time, DemonWare had already helped with a slew of games including Call of Duty 2: Big Red One and Call of Duty 3 and a variety of titles from Namco, Sega, THQ and Ubisoft. Once the purchase went through, DemonWare finished out their existing contracts and began refocusing their efforts entirely on Activision games.

Pat Griffith, Activision's vice president of online technology, said that they began working with the company initially to support matchmaking and storage systems needs on the PlayStation 2 and PlayStation 3. But it was their work on Call of Duty: Big Red One and Call of Duty 3 that won the publisher over.

"DemonWare's level of collaboration with Treyarch on Call of Duty: Big Red One and Call of Duty 3 surpassed anything we'd ever seen before," Griffith said. "We found the crew at DemonWare to be highly capable. DemonWare quickly demonstrated an exceptional level of knowledge and experience with our game engine and practices in addition to their domain-specific expertise."

In absorbing the company, Griffith said Activision hoped that DemonWare could help improve Activision's online gaming for all platforms.

While that acquisition may have been the most important purchase the company made in a history fueled by pop-culture-defining games and billion-dollar franchises, it certainly must not have looked that way to everyone involved just months after the deal was announced.

Modern Warfare's Perfect Storm of Failure

Much of what DemonWare learned, much of why Call of Duty games tend to have smooth online launches, came from one disastrous online launch: Call of Duty 4: Modern Warfare.

"The launch of Modern Warfare came at a point where there was a lot more adoption of broadband internet connections," Griffith said. "We had, and the industry as a whole probably had, some issues across the board."

Modern Warfare became, as Griffith put it, a victim of its own success.

"We became more successful than we had prepared for," he said. "That's not just in the online world. Our sales of Modern Warfare were above our expectations."

The PlayStation 3 and Xbox 360 versions of the game sold about 2 million copies by December 2007. By January 2008, 7 million copies were sold worldwide. By May, Activision had sold 13 million copies of the game.

But the publisher, the developers and DemonWare weren't prepared for those kind of sales, and the online projections they use to figure out what level of server support they need is based on game sales projections. To make matters worse, everyone seemed to have underestimated how widely adopted broadband internet would be by mid-2008.

"After the Christmas of that launch we had a very rough time"

"Not only were more people buying Modern Warfare than we expected, but more people were connected that were buying Modern Warfare than we expected," Griffith said. "After the Christmas of that launch we had a very rough time."

Things were breaking faster than DemonWare could fix them, and often, downed systems would have a domino effect on other parts of the game.

But out of that trial by fire came a number of DemonWare and Activision's core tenets.

"Many of our lessons about measuring and everything else, they come from that," Griffith said. "It provided a foundation that we build on.

"I think that the nascence of our philosophy begins in that Modern Warfare game and what happened in that title."

Lessons Learned

DemonWare has several key rules about how to make an online game that stays online, but the most important, it seems, is communication.

Instead of just sitting on the sidelines while a title is being developed, DemonWare's team of more than 150 people stays in close communication with game developers, and helps shape how a game will launch and what online features it will include.

"Before a title is greenlit by Activision we get involved," DemonWare's Kirk said. "What are their online plans? Are they feasible? Are they being overly ambitious for the technology and the size of the title, etc. And that all just rolls in, keeps rolling on through development."

Over the years, DemonWare has compiled a set of online tools and services like stat tracking or leaderboard services that any Activision game can tap into, and the engineers at the company make sure to use them across games whenever possible.

Sometimes they take those tools and customize them. And finally, if neither of those approaches work, they set about designing new online tools.

All of this comes through weekly, daily, even "minutely" calls and in-person meetings, Kirk said.

Their key goal is to make sure that whatever the game includes won't break the online experience.

Once the services are integrated, DemonWare launches into testing, trying to do whatever it can to break what it just helped create.

Another core lesson DemonWare learned from the Modern Warfare launch was to make sure that when something breaks, it doesn't take anything else down with it. Instead, DemonWare has the ability to turn off many online features on the fly.

"One of the things that can happen in any online system is if one bit of the system goes down it can take the whole system down," Griffith said. "I don't think it's magic, I don't think it's anything that nobody knows, it's just very hard to try and decouple and almost isolate a group of systems so that one will not have a knock-on effect on others."

Take Ghosts, for instance, Kirk said.

"Only a small subset of those are required to play a game," he said. "If player counts or the service that creates the global heatmap, if those systems are down, they're down but they're not affecting gameplay. People can still get their profiles, matchmake and join a game. That's what we focus on, making sure those core systems get a lot more attention and a lot more love then the ancillary systems."

Not every online bell and whistle that a developer might want to add will always go into an online game. DemonWare can, if needed, cut online features, Kirk said.

That cooperative decision is based almost entirely on what sort of impact the engineers think the feature will have on online stability.

"We have a lot of smart people who will basically boil the feature down," Kirk said. "We look at what data needs to be stored. We look at how often it needs to be accessed and how frequently it's going to be used. And then depending on what kind of back-end system we select, is there a database we can scale? Is it actually a tractable problem or are there things in here which are just going to cause us problems down the line?"

If it's an easy, scalable feature to solve for, then it more than likely goes in. If not, then they set to work solving the problem.

"Then we start to worry a little bit and maybe start to help the studio to redesign things," Kirk said. "We work together to throw out some different ideas about how we can make it safer and more likely to succeed when the title actually launches."

And it's those problems that the engineers — the technologists, as Kirk calls them — live for.

"We don't just want to spend the rest of our lives doing matchmaking," he said. "Whatever those new features may be, be they dedicated servers or branching out into data warehousing or things like that, then we're always up for the challenge."

Cutting a feature, while possible, is very rare, Kirk said.

"We sit down with studios and there's lots of whiteboards and they'll basically spell out the idea," Kirk said of the process used to avoid feature cuts. "If we look back at the past Black Ops titles, the idea of recording every single game that was played seemed like absolute craziness to us. But we work it through and we work out systems that allow it to happen. One way or another it generally happens, maybe not in the way it was initially conceived by the studio, but a lot of time those things will move forward."

Essentially, when creating a game with an online component, DemonWare and the Activision developers treat bandwidth like any other resource.

"There are CPU budgets for rendering and for AI and for sound and so we're careful how we apportion that out too," Kirk said.

Griffith calls the process "nudging" rather than pure editorial cutting.

"With that collaboration there are a lot of times where something may be a little bit dangerous or using too much of a connection or have the potential to run out of control if it's heavily used or abused, possibly by a player or the software developers themselves," he said. "The game clients can be written in a way where they may do things that can cause runaway failure.

"Keeping the game running is of the utmost importance, so we may make determinations about how we use that very valuable connection to the consumer's console in his home. Which may mean we may not 100 percent do every pie-in-the-sky thing that we want to do because we may want that traffic, that bandwidth reserved for that core game feature. Doing something totally ludicrous that would totally bite into that connection we would probably rule it out."

But that big, ludicrous, game-breaking idea has yet to happen, Griffith said. The team has always found a solution.

More service than game

In many ways, perhaps more than most games, Call of Duty seems to be becoming more of a service than a series of standalone titles. It is, to many, like television, a form of entertainment that you expect to always work and always be ready to entertain you.

Activision and DemonWare don't have to support just a single Call of Duty game either. Currently five of the top 20 Xbox Live titles are Call of Duty games; that's tens of millions of players beating on DemonWare's technology at the same time, said Daniel Suarez, vice president of production on Call of Duty.

"It is for us more than just a game, it impacts pop culture," Suarez said. "People see this as an outlet to relax, to socialize. So for us it is more than that. To us, it is our business, it is our livelihood and we owe that to our fans. We owe that it be up and running and delivering at the highest possible level of operation and that's what these guys do. That's what our developers do. That's what Infinity Ward does. It's what Treyarch does.

"It's not launching the game; it's a live operation 24/7 and we're constantly doing updates on these titles to make that experience better. That's what it's all about for us."

Griffith sees it as both a game and a service. There is at Call of Duty's core the twitch gameplay and adrenaline of running and gunning, but if you can't get into the game, if you can't start the game online, then none of that matters.

"If you can't talk about, share about or see what's going on with the rudimentary game, the other stuff is superfluous," he said.

That so many companies seem to have problems getting players online and keeping them there, is more about Activision and DemonWare's past failures than anything else. Call of Duty failed spectacularly first, so these future iterations are now more stable. That other games, like Battlefield 4, are now going through their trial by fire, is more about timing than anything else.

"I think there is probably a lot of experiences that they're going to gain," Griffith said. "Our online games have been maturing with the online community and expanding with the online community. So we've learned some hard lessons. Other people are learning some hard lessons now. We don't want to repeat mistakes, we want to make new mistakes, I guess. But we try really hard not to make new mistakes either. Every lesson we've learned is baked into everything DemonWare does and everything DemonWare does to collaborate with the game teams. I think that there are growing pains for everyone, it's hard to jump into the deep end of any pool. We do see things we would have done differently but they came at the cost of experience."

And Suarez points out that Activision and DemonWare are still learning.

"We still make mistakes," he said. "There are still things that break. It may not be as transparent to the end user, but those things happen. We consider ourselves fortunate that we're doing the planning and have the right people to do that, but we are far from perfect and far from mastering this. We're still learning as these new platforms come out. It's all constantly changing as we go."

A sense of dread

For DemonWare, the sense of dread that comes with the launch of an online game never really goes away, it just subsides a little.

"No matter how much preparation that we do there is always a nervousness and a tension around launch day," DemonWare's Kirk said. "Even though we test as much as we can, different behaviors will just emerge when real people start playing the game. We just have to be ready to react for that. We have solutions for a lot of the common problems, thankfully, because we employ [situational monitoring] systems on the backend. We know we can scale things up pretty easily. If we need a new database or more read capacity or more write capacity, we know how to do it and it's a well-practiced drill."

Those launch day jitters make way for DLC launch concerns and then worries about the initial flurry of title updates. Throughout it all, DemonWare engineers are watching graphs and charting behavior 24 hours a day, seven days a week. Eventually, those once-new patterns start to normalize and the folks behind the endless sessions of online Call of Duty can relax, if just for awhile, until the next Call of Duty hits.

All of that nervousness and effort is driven by the knowledge that gamers expect their online games to be as reliable as buying something on Amazon or watching a movie on Netflix, Suarez said.

"I think the Call of Duty consumer, when they put the game in and have an hour of time to play, they want to have a great time," he said. "That's what we offer our customers back and that's what we work so hard on. We know at the end of the day that's what they expect. We make that investment in infrastructure because it's critical to our business."

And investing early on in DemonWare, Griffith said, is a crucial piece of how Activision is able to launch games online with less hiccups.

"I believe that DemonWare is crucial to the success of our online games for all of the reasons mentioned previously," he said. "They give us a stable platform on which to build. They provide us with valuable data about how our games are played. And their collaboration with the game development studios like Infinity Ward and Treyarch has allowed our game teams to focus on making great games."