This is what it's like showing an unfinished game at PAX
How games are made: Defense Grid 2, part three
I
In four days Hidden Path will show its unfinished game, Defense Grid 2, to the public. It will show a new multiplayer level, the first it's ever tried with Defense Grid. It will be the first time anyone outside of Hidden Path has ever seen it. And it's not yet ready.
HPE CEO Jeff Pobst and Marketing Director Shannon Gerritzen just arrived back from Gamescom in Germany, where they showed a few dozen press outlets the single-player "first playable" level of DG2 — the first half of Marketing Plan A. While they were away, the second half of the plan rested in the hands of a small team of developers that is, at this moment, frantically trying to pull its end together.
Later, Pobst characterizes the mood as "normal, but time-intensified." It's not quite panic, but it's close. Ask any one of the game's developers and nine out of 10 will tell you their single biggest concern is the multiplayer level. Since development began on DG2, the multiplayer has been the one great unknown, the big, black mark on the calendar that every team member has dreaded having to face. And now it is here.
The technology took four months to build. The level design has had four days. And in those four days, Lead Designer John Daud has come to a harrowing conclusion: The game isn't fun.
PAX begins on Friday. The final deadline is Wednesday. It is now Monday.
This is game development.
Spawn at Death
The Defense Grid 2 team has grown since my last visit. There is now an extra pair of workstations where before there was only empty space. Wires snake along the floor, covered in wide sheets of thin plastic.
It's still morning when I arrive, but the team has been in motion for hours. The small studio is filled with computers, equipment and supplies for the PAX excursion. People are in motion. Talking. Occasionally running.
Gerritzen and Community Manager Christa Charter huddle in Gerritzen's office, preparing a presentation for the entire company about Gamescom and PAX. The studio has, on previous visits, exuded the air of quiet, calm confidence not unlike that of a surgical bay. Now it feels wrapped in tension.
And there's another first: Pobst's door, almost always open, is closed. Inside, he and Daud are in deep discussion, charting graphs and curves on a whiteboard. I will learn later that they are attempting to quantify "fun."
Daud started work on the multiplayer level practically the same day Pobst and Gerritzen left for Germany — not by choice, but through circumstance. The game as a whole had to be playable before it could be made to be "multiplayable," and the "first playable" was finished just in time for Gamescom. This left Daud only nine days to finish his work for PAX after finishing the final Gamescom build.
While Pobst and Gerritzen were in Germany, Daud was building, testing, tweaking and re-building. After a few days of work, he started playtesting with members of other HPE teams.
"I want to say they were having fun, but it was reserved," Daud says. "It wasn't boisterous yet. ... Up until that point, we hadn't actually played multiplayer. ... When you actually started playing and thinking about how this game mechanic is playing out, strategies changed. It was a bit alarming."
Any given DG2 level is basically a playing field. Enemies (called "mobs" in developer shorthand) enter from one end of the field, traverse to the power cores on the other end, grab the cores, then walk back out again. In between the mob entrance and the cores is a grid upon which the player can place defensive towers to try and kill the mobs before they escape.
In initial playtests, Spawn at Death proved fun, but there was something wrong with it.
In the single-player game, the player's goal is to kill the mobs as fast as they can, save the cores and also score points. In multiplayer, two players are playing a single-player level against one another, so the end goals are reversed: The winner is the player who scores the most points. Killing enemies is just one way to do that. Saving the cores — well, only if there are points in it. It's a mercenary, win-at-all costs variant of a game that, for nearly eight years, has never endured such a dynamic shift.
Whether or not this is fun to play depends on a multitude of variables, which Daud had guessed at in his exhaustive design documents, but that he hadn't been able to test until a handful of days ago. Variables like how much damage each tower deals, how much health each enemy has, how those two lines intersect and how many points are awarded for which things. Tweaking this math to make the intersection of those lines fun is the essence of game design, and for DG2, that job falls to Daud.
"We play one game all day, and usually one 30-second segment of that game, repeatedly until it's right," Daud says. "We're not playing games all day. It's one game, one level, one stretch of road, whatever until it's right. Then you move on to the next part, and maybe that breaks the part that you just spent a day playing over and over again. It's a lot of repetition. It's a lot of minor changes that may have cascading effects.
"Rinse and repeat ... is what we do, until it's good or until someone says, 'Enough.'"
The core of the design for DG2 multiplayer that Daud has been rinsing and repeating for a handful of days is what's called "Spawn at Death." Mobs killed by one player vanish from that player's map and get placed ("spawned") on the opposing player's map, but they're stronger than before. This is how the two players will interact. They aren't shooting at one another, as in most multiplayer games; they're each defending their own grid and their own power cores and placing their own towers to shoot their own mobs. But the better they do, the harder they make it for the other guy. It's a race, in effect, and the player with the most points at the end wins.
The hitch is that the interaction is subtle. If you don't already know what's happening, you won't figure it out by watching the screen. Mobs spawn in the midst of other mobs. They just look like more enemies to kill, and taken as a whole, the multiplayer level plays almost like a single-player level. It may be a little more fast-paced, but you don't "feel" the presence of the other player.
Daud's plan to address this: Spawn at Death. Instead of spawning slain mobs at the level's entrance, like usual, he wants to spawn them at their place of death, but on the other player's map. So if you and I are playing and I kill a mob near my power cores, for example, that mob will spawn on your map near your power cores, but it'll be stronger.
In initial playtests, Spawn at Death proved fun, but there was something wrong with it. Daud suspected it could be tweaked but also began to doubt whether or not it was the right way to go at all. With less than a week to go before PAX, he called a meeting with the lead designers of HPE's other projects and HPE CTO Michael Austin, and showed them his design. They hated it.
"When it was presented," Daud says, "verbally ... the reaction was, 'Oh, no, that's not good. You could break it!'"
Daud's colleagues spotted the design flaw immediately. Players could simply concentrate fire near their own power cores, forcing their opponents to deal with waves of stronger and stronger mobs in prime position to grab a core and run away with it. Mobs spawning near the cores wouldn't have to run the entire gauntlet of towers (and subsequently absorb damage). They could start stronger halfway through the there-and-back run and get away. A player who gained a slight advantage early in the game could run away with it, leaving their opponent wondering what even happened.
Fun for the winner. Borderline abusive for the loser.
Daud's colleagues convinced him to remove Spawn at Death and instead spawn mobs at the level entrance, as in the single-player game. Daud agreed, reworked the game and tested it. And it was even less fun.
"In theory, this should not be fun," says Daud. "And all of my peers are telling me this is broken. And yet the experience is actually fun."
Daud was stupefied. And right back where he started. And then it was Monday.
Monday
"When I came in Monday morning, I got an email from Dacey," says Daud, "that basically said, 'Design ideas are coming. Warning!'"
Producer Dacey Willoughby had been detailing the struggle with Spawn at Death to Pobst in Germany. And Pobst and Austin had been burning the fiber wires connecting Europe and North America, brainstorming ways to fix it.
"When Michael and Jeff get going, it's just a ton of ideas," says Daud.
Those ideas arrive via email a short time later, shortly before Pobst walks in the door. And they're good ideas. Potentially game-saving ideas. The challenge is figuring out which of them can be implemented in less than four days.
What Daud and Pobst have realized is that Spawn At Death adds interaction to the core Defense Grid experience, which they knew, but it also adds something they didn't plan on: tension.
"What was fun about Spawn At Death [is] it was fast," Daud says. "It was furious. That made it compelling."
They decided to stick with Spawn at Death but to focus on how to ratchet up that tension and make it obvious to the player what, exactly, is happening. As for the possibility that players may "exploit" the ability to spawn stronger mobs near their opponents' power cores — well, it's a four-minute demo. The hope is that they can tweak the fun and reduce the impact of that flaw, and maybe sometime after PAX, eradicate it entirely.
Pobst and Daud are discussing the new ideas behind closed doors on Monday. There are "a few features" that Pobst hopes will enhance the fun. After the closed-door huddle, their conversation moves to the bullpen, where Pobst outlines to the rest of the team the elements of the game design he wants tweaked. The question is how to tweak them and how much.
Math, in essence, that has to be tweaked and tested, rinsed and repeated, until the balance is perfectly fun (or, at least, fun enough for PAX).
Lead Engineer Matt "Twig" Johnson and his engineers are tasked with implementing sliders into the level editor to allow Daud to change various settings on the fly — things like the number of points awarded for kills, saving power cores and placing towers, the health values of mobs and how long its takes for them to spawn. In essence, it's math that has to be tweaked and tested, rinsed and repeated, until the balance is perfectly fun (or, at least, fun enough for PAX).
Next, Lead Artist Lex Story and his artists must create creating new effects to inform the player that things are happening and why. They need a particle effect that shows mobs spawning at death and a few other bits and bobs of art to make the whole thing feel intentional and not like a glitch.
Meanwhile Willoughby, quietly observing, keeps track of it all, adjusting the production calendar on the fly to accommodate the changes. Each minor tweak, by itself, may be unnoticeable, but the hope is that, in concert, they will radically change the game. Really, they have to. Plan B is to show the single-player level instead of the multiplayer. But that would be a failure. HPE needs to show the multiplayer. Its research has shown its players want multiplayer. It desperately wants to show them the multiplayer. And PAX is the perfect place to do it. The team has two more days until the game is locked.
Floating trees
Weeks before this last-minute scramble for fun, a decision was made that would nearly wreck the whole thing. Nobody knew it at the time, but it would have lasting repercussions. It wasn't even a big thing, really. It was just one small decision in a long line of them, this one about shadows.
Shadows perform multiple roles in a game like Defense Grid 2. The most obvious is that they add realism. In real life, big things like towers and trees cast shadows. So, in a video game, if those things aren't doing that, they look fake. More importantly to DG2, shadows also impart scale.
The towers in DG2 are designed to be huge, but the players are viewing them from a top-down perspective. In real life, when you're standing on a tall building and looking down, everything below will look small — even cars and trucks and other buildings. Flying high in an airplane, even the tall buildings look small. It's a matter of perspective.
DG2 is played from airplane level. So unless you zoom way in, everything on the level looks small. Even the towers. That's where shadows come in. Big trees cast shadows. Bigger towers cast bigger shadows. In DG2, the towers are supposed to be massive, rising high above the treetops. So they need massive shadows.
But shadows in video games are tricky things. They're mostly code, but since they impact the overall look of the game, they often need an artist's touch. That's where Lead Artist Lex Story comes in. Story wants to make sure that people are excited to play DG2 and that they are impressed with the visual awesomeness of the towers they're deploying to deal death to the enemies below.
"[If] people don't buy into it, that would suck for the experience," Story says. "And the experience is what I want to drive home. My job as a communicator is to drive home the fact that this is a world you completely understand; there's no questions about it. Have fun.
"I always tell people, I hate subtle. I don't want subtle. I want obvious. I want in your face. I want, 'Wow, I remember that one scene. That was really cool!' It's like when you watch a scene in a movie and it's super dramatic. Do you remember any of the other parts, or do you remember the super dramatic part when Gandalf comes out of the frickin' mountain going downhill and he shines his frickin' light and it blinds all the orcs?"
"I always tell people: I hate subtle. I don't want subtle. I want obvious. I want in your face. I want, 'Wow, I remember that one scene.'"
For DG2, Story's brand of awesome is for the player to be wowed by the sight of the towers rising from the ground and be awestruck by their sense of scale. That's why he needs (among other things) shadows. But right now the shadows he has aren't working. They float.
"When you're a player and you're zoomed in, it looks weird, because it looks like the trees are kind of floating," says Austin. "Shadows are there to ground trees. If the base of the tree is not touching the shadow, it just looks kind of wrong."
Austin is the executive producer on one of HPE's other games, Windborne, but he's consulting with the DG2 team because he's in charge of all of the technology for the entire company. And he wrote most of the engine the team is using, back several years ago, for a game that never came out.
In the real world, shadows are the absence of light. We say that shadows are "cast" but they're really the space where light isn't cast, because something (a tree) is standing between where the light is coming from (the sky), and what it is shining on (the ground).
In a video game shadows are created as a separate system from light, because the light is itself just another visual effect. It isn't real light; it's an illusion of light. Video game light doesn't cast shadows. It doesn't make things look brighter. It doesn't do anything, because it isn't anything. It's just a trick. Things look brighter in video games because those things are told, by computer code, to look brighter because, according to the math, those things are where "the light" would be shining on them. Game engines calculate where light would be coming from and where objects would be hit by it and then fake the result.
Shadows are created the same way, except by calculating where light wouldn't hit based on which objects would be in the way. The places in the game world where light wouldn't reach because of another object in the way are where shadows get placed, in the shape of those objects. It's similar to how it works in the real world, but the exact opposite. Shadows in games aren't the absence of light; they're the presence of shadow. It's another trick.
There are many techniques for doing this and making it look realistic. The technique DG2 uses creates shadows that go from solid and dense at the center to lighter and more diffused around the edges to simulate the fuzzy-edged look of real shadows. It looks natural, and it grants the objects in the game a sense of scale. But to make it work in the short time the team members had before PAX, they had to make one fateful decision that will come roaring back to bite them in the ass: They turned off "culling."
Culling is a method the game engine uses to determine what elements on the screen the player can't see, and then it decides not draw those things in order to save resources. If a player can't see a thing because it's blocked by something else, or is off screen, then there's no need to draw it. Why would there be? You can't see it. So those things a player can't see are "culled." Austin compares this to the door to a room down a hall beyond our line of sight. We know that door is there but we can't see it. In a video game, the team would "cull" that door, and simply not draw it until we could see it, saving precious GPU cycles and increasing the frame rate.
For whatever reason though, culling interferes with the way shadows work in DG2. Engineer Peter Freese recommends against turning culling off and instead working on an alternate way to draw shadows, but he's overruled. It's a small thing, in spite of that. There's plenty of GPU power to go around, and the PAX demo is just for PAX. The shadows can be fixed later. The demo will just have to get along without being culled.
Meanwhile, the shadows are creating a problem people can actually see, if they look closely enough. The shadows in DG2 are fuzzy and soft around all of the edges, including the bottoms where the shadows meet the trunks of the trees. This makes it look like the trees are floating, which is not great.
Unfortunately, they may not be able to fix this in time for PAX. The issue is put on the board for fixing before the game's release, and maybe before PAX, if there's time.
"When it's coming down to the line and you have a big list of things to do, a big task list, spending your time fuzzing out the leaf border of these shadows is maybe not where you should be spending your time," Austin says, laughing. "For the final game it's fine. But it's really easy ... We call it rabbit-holing. It's where you get so focused on the tree that you completely lose track of the forest. ... 'How far do we go on everything to get the best gestalt?' rather than 'How do I make this perfect?'"
The trees may just have to float. It's nearing five o'clock on Monday. For the DG2 team, the day will not end for another several hours. And then there will be only two days left before PAX. The shadows will be put aside and forgotten until Thursday, less than a day before the show starts.
Whiteboard
It's Tuesday afternoon. Pobst is a whirlwind, chasing conversations across the room, checking in with the various projects underway, assigning out tasks and making sure everyone knows what everyone else is doing and how their tasks fit into the whole.
Daud has hacked in a sound effect for the mob spawning to alert players with an audio cue that something has happened, but it's awful and distracting. It doesn't help solve the problem. It may, in fact, make the problem worse, creating noise that the player will ignore.
Pobst tasks engineer Mark Hein with tweaking the way the sound is played and producer Willoughby with finding a better one. Daud, meanwhile, goes back to tweaking numbers. Johnson suggests that the points awarded for saving power cores aren't quite right. After a long conversation, Pobst conditionally agrees. They tweak the points.
Meanwhile the spawning effect still isn't right. Pobst wants it to be longer, to give players more notice that something is about to happen and more opportunity to notice it when it does. Daud disagrees, but relents. The spawning effect will be tweaked from two seconds to three.
Pobst tasks animator Kevin Loza with redrawing the particle effect to make it longer and more robust, even though a bug has developed since last night that makes the effect not even visible. (Someone is working on that.)
In the course of a 45-minute walk around the room, Pobst has checked in with almost every member of the team and influenced almost every aspect of the game still in production. People are moving, working, talking and the game is changing yet again. It is, to borrow Pobst's word, intense.
By the time Pobst leaves the bullpen, the giant white board behind Willoughby's desk has re-populated with a rainbow of task cards reflecting every conversation that has just occurred.
By the time Pobst leaves the bullpen, the giant whiteboard behind Willoughby's desk has re-populated with a rainbow of task cards reflecting every conversation that has just occurred. Willoughby was watching, taking notes and tracking. This impromptu meeting has been recorded in real time, and by the end of the day, most of these tasks will be sorted and the cards repositioned to the "complete" column.
Pobst's "few features" have been implemented and tweaked. After PAX, each of these features will be tweaked again. The particle effect, in particular, will get an update. Daud wants a teleportation effect on both sides of the game, zapping dead mobs upward and over to the other player's map, then beaming them down onto the grid. But for now, those additions will have to wait.
By the end of Tuesday, what the team has planned will have to be "enough." Making sure it's all as good as it can be is now the task. Tomorrow is Wednesday, and by the end of the day, the game will be frozen, compiled and burned to disc for the trip across the bridge to the convention center.
Barring catastrophic bugs, the team will be finished. It will have succeeded in making a playable game in just over six months and turning that game into a multiplayer experience in just under nine days.
Whether it succeeded in making the game fun remains to be seen.
Not showable
The burgers arrive and they're huge. We're at a favorite lunch spot for the Hidden Path staffers. The majority of the team is there: Willoughby, Story, Johnson and a half dozen or so others.
It's Thursday, the day before the doors open at PAX. Several hours ago, late last night, Hidden Path locked the final demo build of DG2. In another 20 or so hours, these same developers will be on the show floor, waiting for the first public showing of the game they've spent the past six months building.
The mood is startlingly light. I begin to think this might go exactly according to plan until my milkshake arrives and a message pings on Johnson's phone.
As the conversation continues around the table, Johnson responds to the message, then gets another, then responds more hurriedly, then excuses himself from the table. When he comes back it's clear something has gone wrong.
At the convention center, Pobst, Gerritzen and Austin are making final preparations at the booth. They install the build of DG2, begin playing and react in horror to what they see: The game is "unshowable."
By the time the DG2 team returns to the studio, developers are already breaking out spare laptops and trying to replicate the setup at the booth. I'm reminded of the scene from the movie Apollo 13, where Gary Sinise sits hunched in a replica of the Apollo command module, flipping switches and cursing. In this version, Willoughby and a producer on another HPE project play the Sinise role, but their flipping of switches fails to reproduce the problem. To them, the game seems fine. Maybe a little sluggish. They wonder if it's a problem with the mouse drivers. Maybe the laptop's power settings.
They alter the settings on the laptop and get a better result. The game seems to run faster. Pobst, over the phone, is still not impressed. There's a latency issue and he believes it's in the graphics code. Austin, also at the convention center, is convinced it's a vertex issue and heads back to the studio to help sort it out.
The problem, he tells me when he arrives, is with the interface. Moving the cursor around the screen, there's a lag. A move of the mouse produces, after a brief, but noticeable delay, a move on the screen. The response is not "glassy" like he wants it. He's convinced there's something happening in the build that's dragging down the graphics processor.
Austin goes to work, and in a matter of minutes he has the DG2 engineering team organized to hunt and kill the problem. He installs a CPU monitoring program on one machine and a GPU (graphical processing unit) program on another. The CPU program finishes its work first, running a brief section of the game and then spitting out a report on what's happening in the CPU, and why and for how long.
Austin nods his head. The CPU performance is fine, as he expected. The game is "GPU-bound," definitively. It's asking the GPU to do so much that it is falling behind in tasks, and the cursor and UI elements aren't being drawn fast enough. It "feels" slow. A frame rate tool confirms the frame rate (how quickly the GPU draws graphics on the screen) is half of what it should be. And then the GPU tool provides the answer: Every single element in the game is being drawn four separate times. The GPU is, as Austin suspected, vertex-bound.
It's a snowball of incidents that no one could have foreseen, but that has ground the game to a halt. Game development chaos theory. A shadow blurring its edges in one place creating a tsunami of vertex binding in another.
And that's when it dawns on everyone almost simultaneously — the culling. The fateful decision, made weeks ago, to turn off the culling to save the shadows. Everyone knew that this would increase the frame rate, but no one fully understood just how much it would do so.
In order to make the shadow hack work, culling was turned completely off. This means that for DG2 multiplayer, the GPU is drawing one map for player one and another map for player two. Add to this the fact that the team recently added a "picture-in-picture" feature, allowing each player to see what the other player is doing in real-time, and add the fact that the GPU is being asked to run two complete instances of the game simultaneously and draw every single object four times. This is a massive strain, and that's being reflected in the frame rate.
When the decision was made to turn off culling, no one knew they'd later want to add picture-in-picture. And when they added picture-in-picture, they'd forgotten about culling. It's a snowball of incidents that no one could have foreseen, but that has ground the game to a halt. It's game development chaos theory — a shadow blurring its edges in one place creating a tsunami of vertex binding in another.
Austin concocts a plan to turn culling back on, but to draw a bubble around the shadows to prevent the culling from interfering with them. Meanwhile, on each player's machine, the game will only draw what that player should see and not everything else. In less than an hour after Austin's return, the plan is in motion. Austin has diagnosed, tested, proven and set about making a fix to a complex, hidden graphical problem in less time than many of us take to make breakfast.
It is as if, riding on his white horse, he crested the hill just in the nick of time, staff (of smarts) held aloft, dazzling the orcs (of graphical glitches) and bringing with him the dawn. If I hadn't seen it for myself, I would have doubted it. And if I hadn't seen the mounting panic and puzzlement in the faces of the dozen or so younger, less experienced developers, I would have doubted it wasn't a ploy for the cameras. But these "younger" developers are all industry veterans. The newest member of the team has been doing this for 17 years, and only one or two are fresh out of school. The rest have been developing games for a decade or more, and Austin just took them to school.
This is what Austin used to do a long time ago as part of the "Advanced Technology Group" at Xbox. He and the rest of the ATG team led by Pobst would fly into a studio on short notice, arrive in a rush, diagnose problems quickly and then solve them definitively, ensuring that games made for Xbox were the best they could be.
When I ask Austin if, in solving the culling problem, he's flexing his ATG muscles, he looks puzzled. He thinks about it for longer than he took to correctly guess what was wrong with DG2 and then smiles and says he guesses so, but he never would have thought about it. Then he hurries away to get back to work on Windborne.
When the fix is in less than an hour later, it works. The frame rate has more than doubled. Michael Austin has saved the day.
The first wave
It's Friday morning and Gerritzen has found the cool spot. It's just between the prize wheels.
Players who visit the booth to play Hidden Path's games will be asked to complete a survey afterwards. Those who comply will get a token which they can redeem for a chance to spin the wheel and win one of various prizes: a collectible pin, a poster, a t-shirt, maybe a card containing a beta key for the game. There are varying amounts of each.
Pobst worked out "a simple equation" to determine how many spots on the wheel should be allocated to each prize, and in which positions, based on how many of each prize they have and how quickly they expect to hand them out. If his equation holds, they'll run out of each different prize at roughly the same time. If it doesn't, well, this is a challenge he set himself to distract from the greater challenge over which he has no control: how well people will like his game. The math for the prize equation is simple to Pobst. He's a rocket scientist. To him, that math is the equivalent of a doodle you'd draw while on the phone.
The prizes are loaded into bags. The wheels are placed on podiums. Behind the wheels, there are sixteen Alienware gaming laptops in two rows loaded with Windborne on one side and DG2 on the other.
Slightly in front of these rows of laptops, directly behind the prize wheel podiums and squarely on the center line of the entire booth is the cool spot, where Gerritzen now stands. Throughout the preceding three days of setup, the air conditioning in the expo hall has been strangely absent. It's been stifling. Today for the first time at the start of the show, there's airflow in the convention center and one of the massive vents is dropping a river of cooled air right in the center of the HPE booth. Gerritzen is standing directly beneath it, smiling. It's the happiest she's looked in days.
This is HPE's first ever conference booth. It's smaller than a Manhattan apartment, but it's organized and assembled so efficiently, you'd never know the team had never done it before. Mostly due to Gerritzen.
Over the past few days, Gerritzen and Pobst (and a few helpers) have overseen the expo crews which installed giant flat screen televisions, loaded last-minute builds of each game onto the laptops and installed them in the booth.
The a/v set-up in the booth is tweaked and tested. Speakers are hung. Volume settings are adjusted. The podiums are slightly repositioned. The rope lines are strung. The games, most importantly, are running. And then — suddenly it seems — after six months of development, nine days of work on the multiplayer game and several straight hours of more or less controlled panic, there's nothing left to do. Hidden Path is about to reveal its work to the world. All that remains is to see if the world will show up.
Actually, we are aware the world has shown up because we can hear them just outside, roaring and cheering and clapping in unison as they are warmed up by the Enforcers in the "queue room." Tens of thousands of PAX attendees are queued up in snaking lines, packed in like sardines, each wanting to be the first to get inside. Most know they won't be, but they don't care. The volume is astounding.
The real question for HPE is whether or not any of these cheering, stomping people will stop in its booth. The wait won't be long. At exactly 10 a.m., the doors to the expo hall open and they come flooding down the aisles in a wave. You see them coming and instinctively panic. You can almost feel them pressing air in front of them, like a subway train.
Before the awesome spectacle even fully registers, there's a line at the HPE booth. Over a dozen people are waiting to play DG2. A dozen more are waiting to try Windborne. And then, for the HPE crew, it's time to go to work.
Even TotalBisciut, the cynical, British YouTube star seems to enjoy himself playing DG2.
First up is a woman in a wheelchair with a bulging backpack hanging off the rear. The HPE booth was designed to be wheelchair-friendly with two stations set up specifically at wheelchair height. But no one thought of backpacks. The passageway is too narrow. An interim solution is cobbled together while Pobst and members of the Windborne team reposition the podiums once more, making the passageway wider.
"It's theater," Pobst says, referring to the strain of waiting, then the last-minute scramble to adjust once the show is underway. And he's right. After months spent making a game, HPE must now tackle the completely separate project that is showing the game, and it is no less challenging.
Ten minutes after the start of the show, the lines for both games are full and players are cycling through, playing the games, filling out surveys, spinning the wheels. Both prize-givers find they're giving out almost exactly the same number of each kind of prizes at exactly the same time. Pobst's math works flawlessly. His game, less so.
Barely five minutes into the day, one of the DG2 machines crashes. The machines are paired, so that one player will be playing the multiplayer level against another player. The game is supposed to wait for both players to "press any key" before starting, but one stubborn pair of machines refuses to cooperate. The game periodically starts running before both players have pressed any key, kicking off a solo experience for one player while leaving the other hanging.
John Daud is manning the DG2 play stations. He doesn't know what the problem is or how to fix it. For now he's simply restarting the game each time and carrying on.
"I don't want to think about it," he says. So he doesn't. The problem is intermittent and decreases in frequency throughout the day. It is most likely simply a ghost in one or the other laptops, or it's some flaw in the networking cards. Something Daud couldn't fix even if he wanted to.
What he can fix is a problem with the level timing. In the rush to correct the graphical frame rate, the latest build was assembled to run an older game script. The script is what tells the game when to make certain things happen, like raising a new part of the level out of the sea or spawning certain mob types. It's the latter that frustrates Daud. The newest, coolest mob is a giant, hulking enemy called the juggernaut, and with the old script running, the juggernaut doesn't appear until it's almost time for the demo to end. Most players aren't even seeing it.
Daud needs to wait for a break so he can adjust the script on each machine. It's a simple tweak, but it will take a few minutes, and at the moment, the parade of players is non-stop. And they're enjoying themselves.
Some players have no idea how to play and simply scroll around the world, looking at the scenery while mobs storm in and whisk off their power cores. Other players get hung up on how to use one tower or another, mainly the boost, which requires them to build another tower on top of it. Most players, however, dive right in and compete in earnest against their opponents, smiling, occasionally laughing and frequently circling back around to get back in line for another go.
Even TotalBiscuit, the cynical, British YouTube star seems to enjoy himself playing DG2. (He'll later call the game "pretty damn cool.") By noon of day one, you'd never know anyone was in the least bit concerned. The word "success" feels entirely appropriate.
Aftermath
It is now Monday again. It's been a week since I arrived at Hidden Path to cover its road to PAX. It's been barely two weeks since work began on the DG2 multiplayer level and less than four days since PAX began.
Yet, walking into the HPE booth, the booth is running like it's been in business for years. The rope lines have been streamlined, the podiums moved again to clear more space and the lines still continuously filled. The booth is running like a machine, and in spite of the generally high level of exhaustion, the developers of DG2 seem like they're on top of the world.
Sixteen laptops running Windborne and DG2 have been cycling through their respective four-minute demos for eight full hours each day for four days. In all that time there has been only one brief lull during which there were no players and there was no line. This lasted for, perhaps, five minutes on Saturday. That same day, toward the beginning, there was a line stretching entirely around the booth.
For most of the rest of the time, attendance has been somewhere between those two extremes. Usually there are just enough players waiting to fill both rope lines, cycling through patiently, playing their demos, filling out their surveys, receiving their tokens, spinning their wheels, accepting their prizes. And then, in many cases, they're getting back in line again to do it all over.
On Saturday morning, Daud swapped out the DG2 scripts, ensuring that the juggernauts arrive on time. From that point on, the DG2 machines have been running the modified multiplayer script, in which the level transforms about halfway through and the new, awesome juggernaut trundles toward the power cores in time for most players to see it (and maybe kill it).
The bug affecting the pair of machines that would occasionally refuse to load the game has been fixed. Johnson worked overtime Friday night to rush that and other bug fixes into a new game build, which was also installed on Saturday.
The result, says Daud, is a much more polished experience for players. He believes he can see the difference in how much players are enjoying it versus what he saw on Friday. He says they like it better already, even with small adjustments. He may simply be high on sleep dep and the euphoria of having neared the finish line, but to me it looks like players are having a better time as well.
And Daud also looks better. On Friday, he scanned the growing crowd with apprehension in his eyes. He looked overwhelmed and a little lost. He complained a little about the volume of the booth's speakers and looked like he wanted to play the game for everyone instead of letting them experience it for themselves.
I've never broken down 16 game stations in 15 minutes, and I'm not sure anyone else has either.
Today, after getting one group of eight players running on the DG2 demo machines, he steps back to the line to introduce those waiting to the concepts of Defense Grid, setting their expectations and introducing those who aren't familiar to tower defense. He answers questions. He talks to fans. He works the crowd like he was born to this, and the crowd responds. This, too, may be why they're having more fun.
In just under nine days, DG2 went from single-player to multiplayer, and in just under four, John Daud went from game designer to showman. He's looking forward to breaking down the booth, getting some sleep and then going back to being a game designer again on Tuesday morning. He wonders if he'll remember what he's learned.
The math Pobst worked up to govern the distribution of prizes turns out to be accurate, almost to the minute. Windborne has done slightly less business overall than DG2, and that team still has leftover collectible pins. DG2 runs completely out mere minutes from the end of the show. Pobst seems annoyed that this seems remarkable.
"You take the amount of hours we're open," he says. "Divide by 60. Then divide by five. This is not rocket science."
Nevertheless, it works. The show will close without anyone ever being told "we don't have that." And (almost) everyone will go away happy. (One woman stormed away after learning the giveaways involved chance.)
Just minutes from closing, Gerritzen learns that the Hidden Path booth will be the first one broken down. The crew will arrive at exactly 15 minutes after closing. The area will need to be clear of customers and the furniture will need to be stripped of equipment by the time they arrive.
I've never broken down 16 game stations in 15 minutes, and I'm not sure anyone else has either. It sounds insane to me, but the HPE crew just nods, takes it in stride and begins closing the rope lines and hurrying the remaining visitors through their demos.
At exactly 6 p.m., an announcement declares the show over and the expo hall closed — all visitors must make their way to the exits. This is followed by a sweep of the floor by PAX Enforcers. Fifteen minutes later, the disassembly crew shows up to start pulling things apart, and by the time it arrives the laptops are cleared, the Rubbermaid bins have come out and all of HPE's equipment is stowed or in the process of being stowed away.
What took three days to assemble will be torn apart in 30 minutes and completely cleared away in 90.
The HPE crew stacks and stows and then trundles its belongings out the door to waiting vehicles. And just like that, its PAX is over. Six months plus nine days plus four and then done. DG2 multiplayer was shown. People played it. People liked it. Lessons were learned.
Tomorrow morning Pobst will call the team back together to do a quick recap, itemize a task list, fill up the whiteboard and start all over again. The clock is now reset. The next major milestone: Finish the game.
One level down, over a dozen more to go. The team has less than six months.
This is game development. 
The story continues in Part four.
This story is part of a series covering the development of Defense Grid 2. To read previous installments, please visit the Making of Defense Grid 2 page. This series will continue into mid-2014, the projected launch date for Defense Grid 2.
Editing: Matt Leone
Design / Layout: Warren Schultheis, Matthew Sullivan
Illustrations: Warren Schultheis
Active Discussions
Polygon Quarterly: Not for the Shareholders but for the Games
in Off-topic by sum_guy
Polygon forums close this Friday
in Off-topic by Chris Plante
Looking for more people to write Plight!
in Off-topic by Prinny
Welcome to the >new< forums!
in Off-topic by Prinny
What are you reading manga right now?
in Off-topic by lee87645312