Defense Grid 2 reaches its final stage of development: certification.
[For more than a year, Russ Pitts has been following the development of strategy game Defense Grid 2 through a series of feature stories based on visits to studio Hidden Path Entertainment. This marks the final installment in his 10-part series. Catch up on the whole thing here.]
The screen is a blur of motion.
I'm sitting in the conference room at Hidden Path Entertainment. It is now the end of August, 2014. Defense Grid 2 has been in development for just over 16 months — four months longer than planned. And it's still not finished.
I'm watching the creation of a video that will be released in September. In it, Lead Designer John Daud and artist Chad Haley explain to players of Defense Grid 2 how to use the game's built-in level editor to make custom levels. It is the same tool Daud and Haley use themselves to create levels that will be in the game. They are, in short, giving away their secrets.
On a large television, Daud watches recorded footage of himself playing the game. It will be the backbone of the level editor how-to, with Daud and Haley narrating over it. As Daud watches, he describes what he's seeing, what he did and why. On the screen is a jumble of activity. Lines are drawn, structures created. Daud is watching his own mind in action, seeing himself perform tasks he's performed hundreds of times. At times he struggles to keep up with the video version of himself, realizing after the fact how many tasks he completes without even thinking about them.
Video Daud builds a basic platform and some roads — the building blocks of every Defense Grid level. He creates a path for enemies to walk along, then tweaks the geometry, tapping into years of experience. The decisions seem almost random, but over time, patterns emerge. The ingeniousness of the hundreds of tiny, intricate creations and almost unconsciously made decisions emerges.
As Daud watches the captured echo of himself on the screen, the aesthetics, design and technology of his craft coalesce into a whole. By the video's end, Daud has created a level, a task he has performed over 200 times to date, supported by the creations of technologists and enhanced by creative artists around him, like Haley. Twenty of the levels Daud has designed are in the nearly finished game, Defense Grid 2. And that game is currently in the process of being tested, tweaked and certified.
Defense Grid 2 is less than a month from its planned release date, but for John Daud, the work is done. Here in this room, he's reliving the experience of making it and commentating on his performance to a camera. It is an ex post facto reenactment in miniature of the process he has just endured of making this game that is not yet finished. But his part is done. The game is designed. Now all he has to do, once again, is wait.
This is game development.
Hidden Path Community Manager Emily Zimmer, Lead Designer John Daud and artist Chad Haley
"We planned for it to be hard," says Executive Producer Jeff Pobst, "but we didn't plan for it to be this hard."
Sixteen months ago, the plan was to make "DG2" in less than a year and to ship it only for the PC via Steam. And maybe, but only possibly, on other platforms later if things went well enough. Since then, Hidden Path has partnered with 505 Games to bring Defense Grid 2 to Xbox One and PlayStation 4. That change has allowed Hidden Path to make a better game for more potential gamers. But there's a cost: roughly four months of additional development, give or take, and now, in the game's final stage of development, a rash of headaches.
This week at the company's usually serene offices in Bellevue, Wash., a battle is raging against a seemingly never-ending swarm of bugs. Defense Grid 2 has been in testing for weeks. An external company based in Montreal, Enzyme, has been running the game through a litany of tests, following instructions from Xbox One and PlayStation 4 makers Microsoft and Sony: plugging and unplugging cables, attempting to create weird and seemingly random scenarios and then monitoring the game to see what will happen. The results have been bizarre and confounding, and they have generated a massive list of unexpected bugs that Hidden Path must fix before the console makers will accept the game.
Defense Grid 2 hangs in the air over what will be its final hurdle before the game can be launched. Once the bugs are fixed (enough of them, anyway) the game will be submitted to the console makers for certification, or "cert." Each console has specific cert requirements. Specific language that must appear in help files. Specific ways certain buttons must perform. Expected behaviors when, for example, an ethernet cable comes unplugged in the middle of an attempt to start a multiplayer game in a certain configuration. Never mind that many players use Wi-Fi, because to get a game on a console is to armor it against any and all conceivable errors, no matter how improbable. And it is just these weird, seemingly random "edge cases," that are occupying most of Hidden Path’s time.
"You have two bars and you're always fighting between the two bars," says Pobst. "What I want the customers to have and what Microsoft requires me to have if I'm going to be on their console. Some of the things they require you don't care about, or I don't care about. Certain Microsoft features, supporting this or that. Yes, probably some people care about it, but not as many as the console companies believe care about it. You work on those things too. You try and make it all work."
At the current time there are multiple variations of Defense Grid 2 on Hidden Path's servers: one master codebase, called the "trunk" and three more, called "branches," for each of the three major releases (PC, XB1 and PS4). Each bug fixed introduces new code into either the trunk or one of the branches, and each code change could potentially introduce even more bugs. Some code changes are branch-specific, affecting only one version of the game. But many must be integrated into each version, meaning all of them, separately. And what will happen to each specific version with each new major change is something no one can predict. It’s a process of change, then see.
Each bug identified by Enzyme is vetted in a daily meeting called "bug triage" by Pobst, Daud, Lead Tester Mark Shoemaker, lead programmers Steve Messick and Jason Hail and Producer Dacey Willoughby. Each and every bug report is scrutinized, assigned to a programmer to fix, rejected as inconsequential or sent back for more information. Bugs such as the cursor not behaving as expected for 15 seconds on one level at a certain time must be analyzed as potential symptoms of a larger disease. But some are simply discarded and ignored.
You can almost feel the room lighten when a bug is marked "won't fix." One more step down the road to being finished. One less bug to have to fix before cert. Fewer than a third of the bugs reported get this golden label. Most get fixed, which adds time to an already straining schedule. The game is already behind, and seemingly falling further behind with each passing day.
"Some of the things they require you don't care about, or I don't care about."
The week starts off with a bit of catch-up, with multiple members of the Defense Grid 2 programming team (including Programing Lead Matt "Twig" Johnson) out of the office attending a conference held by engine maker Unity. Those who remain are forced to divvy up Twig's tasks.
Huddled around Producer Dacey Willoughby's desk, programmers Scott Bodenbender, Steve Messick, Mark Heinen and Charlie Ngo "triage" bugs off Twig Johnson's to-do list. It is a long list, but it's manageable. The veteran programmers add the tasks to their own long lists and get to work.
Messick is heading up the Xbox One team, Hail the PlayStation 4. Both were co-leads on the "Chroma" project, a so-called "musical shooter" Hidden Path had been developing with Boston-based music game maker Harmonix. Now that that project is at an end (at least for Hidden Path), Messick, Hail and the rest of that team have been reassigned, mostly to help finish Defense Grid 2. Messick and Hail have worked together at Hidden Path for years. They were previously on the Counter-Strike: Global Offensive team and, before that, worked on various special projects.
Hail spent the weekend finishing a release candidate (RC) for PlayStation 4; a process made more frustrating by world events. Hidden Path happened to be in the process of shipping a certification RC to Sony at the exact same time that a hacker activist group called Lizard Squad was in the process of conducting a denial of service attack against a number of gaming services, including Sony's PlayStation Network. And they were also issuing a bomb threat to an airliner carrying Sony Online Entertainment boss John Smedley.
Hidden Path spent the weekend attempting to test its PlayStation 4 RC and failing, concerned the outages were actually a problem with the game. The RC finally went to Sony late Saturday night, making a long, long week for the already strained developers.
By Monday Hail is back at work on a patch to fix issues Hidden Path knew were in the build but that it didn't think would cause the game to not be certified. The RC comes back from Sony anyway, because of a check box on a form that was or wasn't checked. Sony's representative apologizes to Hidden Path for the red tape, but paperwork is paperwork. The RC doesn't need to be resubmitted, but the paperwork does — for now. No one knows for sure if Sony will accept the game. The weight of not knowing hangs in the air. The game is either finished, on PlayStation 4, or not finished. Schrodinger's build.
In the meantime Hail continues work on his patch. If Sony sends the game back, he'll put in his fixes and send an even better build the next time (along with fixes for whatever else Sony might discover). If the game doesn't come back from Sony, he'll push his fixes as a day-zero patch to be applied after cert. Due to a quirk in the cert process, an RC cert takes about a week, but a cert for a day-zero patch takes about a day. No one understands why. It is just the way it is.
If Hail's week is starting rough, Messick's is even more so. The Xbox One build hasn't even been submitted for cert. Due to the oddities of the Kinect-enabled console, preparing Defense Grid 2 for Xbox One has been unexpectedly laborious, and there have been many more specific requirements for Xbox One than for PlayStation 4. The hope, as of Monday, is that an Xbox One RC will be finished by the end of the day, tested on Tuesday and submitted by Wednesday. Spoiler alert: That's not going to happen.
What the team doesn't know is that an insidious bug has been hiding in the code, for every version of the game, for months. When it emerges, it will grind almost everyone's work to a halt and potentially push the release date for the entire game.
"Due to a quirk in the cert process, an RC cert takes about a week, but a cert for a day-zero patch takes about a day. No one understands why. It is just the way it is."
Defense Grid 2
Not supposed to happen
"At this point, it is out of my hands," Daud says, sitting in a small conference room on Monday afternoon. He repeats the phrase like a mantra, as if consoling himself, or talking himself down from months of feeling as if every single thing was exactly in his hands.
"A lot of the stuff that remains, a lot of the tasks that remain, they're not within my mandate. I can have an opinion on whether a certain bug should be fixed given where we're at, but most of the time I can't fix it. It is kind of out of my hands." Again.
"Occasionally there's something I can do to fix or improve something, but yeah, it's out of my hands, even now." And again.
"I'm just crossing my fingers that it's not going to crash."
But it does crash. Daud himself is the one who finds the bug — on Tuesday.
Daud is in the process of testing the director's cut of level 16 — the giant space elevator mission that Daud originally over-designed, making it too complex and difficult for any but the most experienced deployers. The official version of that level is now much simpler, but as a gift for special edition purchasers, Daud has pulled back some of the changes, restoring the level to how he had originally intended it. He's attempting to test the level when something that isn't supposed to happen happens: A random player joins Daud's game.
The player is most likely a member of the Enzyme testing team in Canada and not someone truly random. But the fact the player is able to join Daud's game is troubling. He's set his game to allow only a very specific type of player. Literally only one, in fact: himself. By selecting to play on a level only he currently owns, the Defense Grid 2 matchmaking system should rule out every other player but Daud. Daud is attempting to connect the two machines sitting on his desk to one another so that he can perform a test, but before he can sign his second machine into his first, the random player appears, drawn into Daud's game by Defense Grid 2's matchmaking system.
I'm sitting in Hidden Path's lounge area when it happens, playing the first few levels of Defense Grid 2 with Pobst. Pobst has been explaining some of the changes compared to previous builds of the game, and he has just paused the game while rolling the credits, long enough to show off the "Thank You" section, in which my own name appears along with every member of the Polygon editorial, video and design teams who has worked on the series of articles about Defense Grid 2 of which this story is a part. It's an odd and touching moment, yet another symptom of the "observer effect." I am shaken from my reverie by the urgency in Daud's voice.
"That's not supposed to happen," he says, and he describes what he's just observed, how a random player joined his supposedly un-joinable game. What happens next is something like panic.
"That's not supposed to happen." ... What happens next is something like panic.
The definition of "any"
After almost a full day of scramble, the bug is identified. Hail and programmers Bob Scott and Scott Bodenbender researched how Defense Grid 2's matchmaking system works. What they found is troubling: it doesn't. The matchmaking is fundamentally flawed, and has been since it was written. But the flaw is so subtle it passed without notice until Daud’s seemingly random event caused it to be revealed.
The matchmaking code was written to solve a specific problem, and it did, so it was called good. But the behavior the code was written to solve might not have been a problem in the first place. The matchmaking system is now behaving in a way that no one intended, because that’s the way it was written.
The problem, Hail discovers, lies with the program's definition of the word "any."
"There's two types of 'any,"' Hail says. "There's 'any' where it means 'don't care,' and there's 'any' where I do care, but I'm letting myself say, 'If there's someone else out there who didn't care then I'm gonna match with them.'"
What Hail is describing is the behavior of Defense Grid 2's matchmaking, the process that finds players for others to play with. The online mode allows players to select from a list of options: which level to play on, what difficulty and so forth. The matchmaking then attempts to find other players who have also selected those same options. If it finds a perfect match — two players who want to play with all of the same options — then there's no problem. If the program doesn't find a perfect match, then it must make a decision about what to do. As of right now, the program is making decisions that, in effect, match any player with potentially any other player no matter what options either has selected, and, in some rare cases, players who have chosen specific options end up in games with options that are not what they've chosen. It is, to use nonprogramming language, borked.
Defense Grid 2 uses a process the programmers call "slackening," to whittle down the list of all available players into a likely match for each other. If it finds a match for a particular option, it moves on to the next, continually narrowing the field of potential matches. If it doesn't find a match for a specific option, then it resets that option to "any" so that it can match a player with someone else who intentionally chose "any." Which is, in theory, what you would want it to do.
If you choose to play on a specific level but can’t find anyone else who has also chosen that level, then you would want to be matched with someone who selected "any" under "level," so that you can get what you want, and since they don’t care, they get what they want too. But here’s where it gets borked: As the code progresses down the list of options, it aggressively asserts its new definition of "any" as "I don’t care" in order to better find a match for every possible player. So a player who chooses a specific map or difficulty may get something they didn't ask for.
This is what happened to Daud. He'd chosen a super-specific set of options, including a level that literally no other player had access to, but since the other box sitting on his desk hadn't yet attempted to find a match using those same settings, he was paired with another player who had selected "any." The game, written to find a match for every available player no matter what, was trying to be helpful. In an effort to reduce the amount of time a player could spend waiting for a match, the matchmaking program was written to force players into a game, whatever their chosen options. But this behavior was not what was intended and, in fact, had been identified months prior by the Defense Grid 2 team and apparently not yet entirely eradicated.
A fix is decided upon: Let people wait. If a player chooses a very specific set of options and can't find a match, the game will create a "lobby" and the player can wait for someone to match them, or else they can start over and try selecting fewer options.
"We're solving [the problem] at the design level," says Pobst.
Work begins on the fix, but another day has come and gone with no end in sight. While the networking bug is being addressed for all builds, the Xbox One version is still struggling under the weight of other showstoppers. It is now the end of Tuesday and there is still no Xbox One RC.
"There's two types of 'any.' There's 'any' where it means 'don't care,' and there's 'any' where I do care, but [someone else doesn't.]"
Lead Designer John Daud and Executive Producer Jeff Pobst
As fast as we can fix them
"It looked like it was within reach Monday and Tuesday morning," Steve Messick says.
Messick was hoping to wrap up work on the Xbox One version on Tuesday and have an RC ready for testing on Wednesday, ready to ship on Thursday. That, for him, would have been perfect. Not only is he leaving for vacation at the end of the week, but Tuesday was his birthday. Wrapping his version of the game would have been a perfect birthday gift to himself. Instead he worked until late into the evening, walking out at around 10 p.m. only to turn around and be back in the office less than 12 hours later. It is now Wednesday and there are still more bugs to fix.
"We've been getting bugs in from Enzyme as fast as we can fix them," he says.
"Right before ship, all you see is all the things you cut at the last minute."
By day's end Wednesday, Messick and his team are still no closer to wrapping the game. Another bug triage reveals another round of critical bugs and Messick's to-do list gets longer. He leaves for home a little earlier in the evening than on Tuesday, but the game is still not in a state that will pass Xbox cert. As a result, tweaks the team would like to make, for the sake of aesthetics or to add new features, are getting left on the table.
"It's always so brutal," says Hidden Path's Director of Technology Michael Austin. "Right before ship, all you see is all the things you cut at the last minute because they didn't quite make it in the time you have.
"You have this picture of what the project could be, and it's beautiful and awesome. Then you go and cut it up and you're like, 'I hope people like the cut-up version.'"
With the week more than half gone, it's still not clear if even the cut-up version of Defense Grid 2 will ship on time.
A happy distraction
The fever breaks on Thursday.
Messick is all smiles when I see him late in the morning. The stream of bugs coming from Enzyme has slowed. What remains are bugs that won't affect cert or else are already being fixed. After 15 months and a torturous week of bug fixing, the end is in sight. The Xbox One RC will finally be sent to Microsoft on the following Wednesday, September 4.
Both the Xbox One and PlayStation 4 versions will need to be certified before the team can fully relax, but for now Hidden Path has done what it can. Work continues on polishing the PC version while the team waits for word from the console manufacturers. Whether the game will meet its September ship date still remains to be seen.
Sitting in the smaller conference room at Hidden Path, Daud explains his state of mind.
"What ends up happening — I think Michael Austin observed this. I had enough not to do that I could fret about little things, little bugs that I couldn't necessarily fix myself," he says. "Everything has — it just wears on you. You see these things and it's like, 'Agh!' Yeah. So."
So. He's making videos and trying to not lose his mind as everyone around him busily finishes what he made.
I ask Daud if he thinks the game, as it will ship, is what he intended. He pauses before answering.
"It's pretty close," he says. And then he launches into a minute-long litany of all of the reasons it actually isn't even close, in his mind. He's not contradicting himself; he's digging deep into what he'd like to see that just won't be in the game. But they're minor details, and most of them were ideas that came about — or were iterated on — during development. The bulk of the game that Daud designed, laid out in painstaking detail in pages of design documents almost two years ago, will be exactly what players will play. And I can see the wonder in his eyes at having pulled that off.
"I can only hope this matches the enthusiasm that I saw surrounding the first game. We'll see."
Even the smallest, simplest games come together as a result of thousands of decisions, each of which could ruin it all. That Daud set out to create this game — and did it — is huge. He didn't make it alone, and couldn't have, but for him, as lead designer, there's a special pride in having shipped it.
Daud mentions how he's still hearing from fans of the first Defense Grid, about how much they enjoyed the game and are looking forward to the next one. Although that does add pressure, he says, it's also a relief.
"To have people compliment that for years is really gratifying," he says. "It's great. I can only hope this matches the enthusiasm that I saw surrounding the first game. We'll see. I hope people are happy with it. That makes it worth it. It has to be fun. I want to make a happy distraction for people."
I ask him if he knows what his next project is going to be, but he doesn't. There are rumors at Hidden Path about what game or games the company is working on next. Some people know; some people don't. And most don't know whether or not I know, so they aren't talking to me about it. But I do know, although I’ve agreed with Hidden Path not to talk publicly about projects that aren’t Defense Grid 2.
I ask Daud about one project, and if he'll be working on it. He thinks he will but isn't sure. He has a meeting right after our interview to find out. So we wrap up and he walks out of the small conference room, and then, some minutes later, into the larger conference room with a small group of others. The door closes and the meeting, to which I am not invited, begins.
On the agenda: what's next for Hidden Path. Michael Austin, Jeff Pobst and Mark Terrano have made a deal with a publisher, sketched out an idea for a game and come up with a plan for how it will be developed. Now, behind that closed door, assignments are handed out, a design document is started and the entire process begins all over again. Some months from now this small meeting will have produced a game. Decisions will be made, code will be written, art will be drawn, scripts, voices, music and schedules will be created, refined, discarded, rebuilt and eventually formed into something you can play. A "happy distraction" that will take you far, far fewer hours to enjoy than were required for its creation.
This meeting is where the journey begins, overlapping where another has come to an end. And the in-between is how video games are made.