The front lines: How a beta makes a game better

Defense Grid 2 enters the final stage of development.

Jump to


More than a year after it started, development on Defense Grid 2 is in the final stage.

It is now May 2014. The Defense Grid 2 team leads at Hidden Path, Associate Producer Dacey Willoughby, Executive Producer Jeff Pobst, Programming Lead Steve Messick, Design Lead John Daud and self-described "dev support guy" Jon Lee are sitting around a table in a room between the kitchen and the rest of the office. There's a Neo Geo arcade cabinet in the corner, board games on shelves against the wall.

This room is usually where the team gathers for fun. This meeting will not be fun.

The Defense Grid 2 beta launched a week before this meeting. Thousands of players have been playing the game. Most of the feedback has been positive. (Too positive, Pobst thinks. He will obsess about the feedback being possibly biased.) But there have also been bug reports.

Beta players are encouraged to send in a report if they discover a problem when playing the beta. Hidden Path has been collecting those, and now has a list. This meeting’s purpose is to sort through them, and make a plan to fix them. If that was all the team had to worry about, the mood would be considerably lighter. It is not.

"This is a very large number of crashes," says Pobst, surveying the list. Portions of the list have been handwritten. It is the most recent tally of "crash" issues reported in the beta. There are around 200 of them.

Pobst wonders if all 200 reported crashes are each unique, meaning 200 separate things that are causing the game to crash. That is yet to be determined, but chances are they are not unique, and that many of the emails are reporting the same thing.

Finding someone to go through the list to determine what is causing each crash, and if they're related, will be tricky. Everyone is busy with other things.

This week there's a milestone for delivering a build of the game to publisher 505. It's also the last week the team has to work on the multiplayer demo being shown at E3 in two weeks. It is also the week all in-game text must be finalized and sent to the localization team that will be translating English into four other languages.

Mention of the E3 multiplayer level causes Daud to shake his head miserably and mutter, "The sync ... the sync."

The multiplayer isn't doing so well. It has a syncing issue, which creates errors and lags when two people are playing together online. It's a typical multiplayer bug, but one that has been nagging the Defense Grid 2 team for months. They now have only a week left to finally fix it.

"This week we want to take care of these people," Pobst says, bringing the conversation back to the beta and the crash reports. "And I'll put up a note and say, 'Hey, this is awesome; we'll give you feedback, but this week we have to be ready for E3. We have to be ready for a milestone. We have to get the voice stuff recording. We gotta be prepared for [localization].'

"This is a tough week to be super responsive to the beta."

This is, in short, the most intense week so far of production on Defense Grid 2, and the strain is starting to show. Voices are sharper, tempers shorter. Faces are lined with worry and lack of sleep. The tension is crowding in on this team like an uninvited guest, and it's going to be around for another two months.

Action items are assigned, the meeting breaks up, and Daud, lumbering to his feet as if the weight of the world were on his shoulders, says he'll be off "making the rest of the game ... in my spare time."

This is game development.

Dg-2 Beta triage with Jon Lee, Dacey Willoughby, Steve Messick, Jeff Pobst, John Daud and Matt Johnson.

A little blip of drama

Simply getting the Defense Grid 2 beta out the door proved to be difficult. The original beta launch date was sometime in November or December of 2013. Then, to avoid the crush of holiday releases and the then-new Xbox One and PlayStation 4 consoles, it was pushed into February. I have the date on my calendar because I was supposed to be present for the event. I arrived, the beta didn't. Luckily (for me and you) we had the subject of making fun to talk about.

By spring 2014 it had become clear the beta would be pushed out again, with the exact release date in flux almost until the very day. By the end of May 2014, Hidden Path and publisher 505 could wait no longer. The beta was scheduled, Pobst sent out an email and the team was put into action.

And then when the day finally came ... the beta didn't go out. Something had gone wrong, and the final build no longer worked.

"The day we were trying to get the beta out ... we should have done a check-in lockdown, where people can't check things in, but we didn't, and so somebody checked something in that broke the game," says Willoughby. "It was right before a release-candidate type thing. That was late in the afternoon, so it was kind of stressful."

At the start of Defense Grid 2's development, Hidden Path was also working on Windborne and the then-unannounced experimental musical shooter, Chroma. Now, with its work on Chroma officially wrapped up in early 2014, it has just Defense Grid 2 and Windborne in the works.

The third of the studio devoted to Chroma has now been diverted to the two remaining projects. Windborne needed staffing up to hit increasingly complex prerelease targets (Hidden Path CTO Michael Austin moved into the bullpen with the team to focus solely on creating the game), and Defense Grid 2 would benefit from the extra hands to beef up testing and problem solving.

But the extra hands would inevitably create hiccups, as developers — not steeped in the intricacies of code created solely for Defense Grid 2 and unaware of long-established and accepted practices — were brought up to speed on the minutiae of a game now in its 13th month of development.

"As we get into the last month of development here, we will have a daily process where we get a build in the morning, and we'll probably have another build by 2 p.m.," says Mark Shoemaker, Hidden Path's lead tester. "We have to make sure we prioritize the issues in it, and demonstrate to the team, and just start working through build churn. A lot of testing is checking the same thing over and over again."

Shoemaker would know. Not only is he the head of testing at Hidden Path; he's also the one who broke the beta.

On the day of the beta release Shoemaker noticed a bug in the audio editor, a sound effect that should have been looping but wasn't. It was a simple "zero to one" flag that, to Shoemaker's eye, had been set in the wrong position. So, unaware that a "code lockdown" had been called, restricting any changes to the game, he made the change.

And all hell broke loose.

"The person that was in charge of the code in that area had gone home for the day," says Shoemaker. "I had fixed a bug ... I didn't even make a code change to the game. I made a flag change in the audio editor that turned looping on for a sound effect. What I didn't realize is that there was no code telling the looping to stop."

The audio file was left to not loop, because of a flaw in the audio engine that hadn't yet been fixed. Shoemaker's "solution" had rendered the beta build unplayable and he didn't even know it.

The looping audio file was discovered during a final play test.

"At the last minute, we were playing the game, and 'Oh, sound is suddenly broken!'" Shoemaker says. "There was this little beehive of activity going on."

What Shoemaker described as a "beehive" has been characterized by some as panic and others as chaos. Whatever the exact definition, it was not good. The beta, which had been working fine earlier in the day, was now broken, and no one knew why.

Shoemaker happened to overhear the commotion and his heart sank. He knew why. He'd broken it.

"I casually overheard somebody say, 'Why is this looping?' And I said, 'I know why that's looping.'"

Shoemaker worked with Programming Lead Steve Messick to revert the change, but the damage had been done. Although the change itself was simple, an entire new build had to be created — and tested — which would take time the team no longer had.

"At the time our builder was taking like 40 minutes to make a build," says Willoughby. "Every time we wanted to get a build candidate, we had to go through that process, and it would be like 40 minutes. When you're getting close to the end of the day, it's like, you almost had it, and then you had to redo something. It was more of a crisis than normal. Fixing the problem didn't take very long, but then rebuilding takes a little while."

The beta was delayed. Again. A release candidate would be complete later that evening, but it wouldn't be released until the next day, roughly 24 hours later than planned.

"Jeff was not happy," says Daud, then begins to say something and stops, gets lost in thought and falls silent.

I ask him, later, what Pobst looks like when he's "not happy."

"There were some raised voices. It's articulate. It's not pounding on the table or anything. It is concise and direct and kinda loud. But not ragey. But you'll know it when it's happening."

"I could tell [Pobst] was frustrated," Shoemaker says. "I felt really bad. But even at the end of it all ... [Pobst] came to me later on and said, 'This stuff happens.' And it does. ... [But] making mistakes is OK. Making them twice is not. If that makes sense."

After the fact, Willoughby would call the event "a little blip of drama." New rules were put in place to shore up the definitions of "code lockdown" and increase communication on the now larger-than-ever Defense Grid 2 team.

"It is concise and direct and kinda loud. But not ragey. But you'll know it when it's happening."

Tester man

The job of managing the bug list falls to Shoemaker. As lead tester, he's in charge of organizing and cataloging those bugs, and organizing the play tests designed to ensure there aren't any more bugs, and so he's gotten the task of applying filters to the data coming out of the beta to help the programmers sort out which bugs to pay attention to, versus which ones might be ignored.

Shoemaker got his start as a QA tester in 1998. As a musician, QA testing was a great gig when his band wasn't touring, and he could easily quit when it was. Eventually he got serious about the job and worked his way up to test lead at now-defunct Surreal Software, a part of Warner Bros. Interactive.

One of Shoemaker's co-workers at Surreal was one Dacey Willoughby.

"She bailed on us to go work for this little tiny company called Hidden Path, working on this tower defense game nobody had heard of," Shoemaker says. "Then ... this email went around one day: Dacey's game came out, this tower defense game, and it's on Steam. Next thing you know, everybody's playing it. We're all playing it. So I got hooked on Defense Grid, and then just through the luck of the draw, I ended up having a chance to talk to Jeff [Pobst]."

Shoemaker got the job.

As Defense Grid 2 gets closer to completion he will also be in charge of overseeing the external testing teams, when the time comes. Since Hidden Path is a small, independent studio, it does not have its own internal testing department. It will be working with publisher 505's QA team and also hiring a group of external testers to do QA on the game.

Shoemaker will coordinate the efforts of those teams with Hidden Path's staffers' work to polish the game and eradicate bugs.

"One of the things I know that's critical with working with anybody outside is access," he says. "Access to communication, feeling like you can touch base with somebody about anything, because a little tiny question will eat at you all day and break your productivity if it doesn't get satisfied in some way. ... When it comes to shipping something, when we're working with outside QA, we have to be organized with what we're asking of them, and we have to make sure that we know how we're going to be utilizing their effort. So that's really a big part of it right there."

Just like many who work at Hidden Path, Shoemaker is an expert in his field. Listening to him talk about how to structure a test program, what to look for in a good tester and how to report data back to a development team is like sitting in on a master class in a field that's so overlooked most people would never assume it could be so complex. It is at once fascinating and a little overwhelming.

He is a man who enjoys breaking things in any way they can possibly be broken, just to find out what happens. To illustrate why he got into QA in the first place, he tells a story about a toy called the "Big Track."

"It was from the 1980s. It looks like a truck. Kind of looks like the tank from Mass Effect. You could program it. My brother is nine years ahead of me. He's interested in programming. He's programming his Big Track to do things. It's got the little dump-truck section on the back to dump toys out or whatever. And he hands it to me, and I'm much younger than him. I'm thinking, 'Oh, this is awesome!' I was 5 or something.

"I just typed a couple numbers into the back, and it immediately goes forward and then turns right to go down the stairs. He goes, 'Nononono!' That was immediately ... our paths were set. He was going to become a programmer and I was going to become a tester.

"'Will it go down stairs?' Yes, it does. I kind of expected to see if it would stop itself or have an error, not going to go there. But it did. It totally went down the stairs. I've been sending software down the stairs ever since."

Defense Grid 2 "It totally went down the stairs. I've been sending software down the stairs ever since."

Defense Grid 2

Jimmy legs

Amidst the sea of chaos currently threatening to drown the Defense Grid 2 team, the one bright spot of calm is, unexpectedly, Art Lead Lex Story.

Against all odds, the art team is wrapped. After redesigning the look of almost every tower and mob after the PAX reveal, Lex Story and his team have moved on to minor bugs fixes and niggling details. They will now either move on to other projects at Hidden Path or else be on standby to jump in on emergency fixes.

"We did a lot of [feature] creep stuff where we would go, 'Hey, you know what would be really cool?' Of course the producer hates that. She gets all wigged out over hearing that kind of banter in the background, without her going, 'Wait a minute; do we have time on the schedule?' I think we've done really well with adding some additional things that we thought were really cool, that we could work with. I... But then we got to a point where, you know what? We've got everything. We just need to create content."

With that done, Story is now doing the one thing he hates most: Waiting. Which leaves him plenty of time for creating ruckus over the change to one of his favorite new objects in the game.

"I called it the master key," Story tells me. The "key" (or, as he sometimes called it, the "boat") is an element added late in Defense Grid 2's development. Intended to serve as an in-game avatar for the player, the "key" would fly into a level at the start of the game and socket into the defense grid, "turning it on" in effect, and giving the player control. Like a key. And also, because the player is theoretically riding in it, a boat.

It is now officially called "the command shuttle."

"They wouldn't tell me that they weren't going to use the name master key, even though I was trying to drum up all this support behind me," says Story. "Everyone would adopt it and carry my standard into battle. 'Yes! It is called the master key!' And then they told me last week ... 'Oh, well, we're just going to let him go on and do his crazy talk and insane gesticulations. We'll get close enough to the end where we'll just drop it on him like a bomb.'

"Pierced my heart with a lawn dart."

Story's current official task is to work on the Defense Grid 2 art book, a giveaway for backers of the Kickstarter campaign.

"It's just amazing how much you put into the thought process and how much you actually show the customer," Story says. "So that's what the artbook is going to do. It's going to basically go, 'Hey, we like you guys as a customer. We appreciate you guys. We want to show you a little bit.'"

What will be next for Story after that is still to be decided. He says he's really "got the jimmy legs" to move on to his next project, but what that project will be is still up in the air. Pobst is currently drumming up additional business, putting deals together for whatever the team (and, presumably, Story) will work on next, which does nothing to decrease the stress at this critical time.

The technical term for this is business development or "biz dev," but it's really just hustle. As an independent developer, Hidden Path has to pay its own bills, and it does this by signing deals with publishers to make specific games, or accepting investment (from crowdsourced backers or money men like Steven Dengler) to publish its own titles. The money that comes in goes to pay developer salaries and cover costs of making a game, and if there's any left over, then that's just cushion for the lean times. Only the very successful or extremely lucky game developers will make enough profit from one game to completely fund another. That's the brass ring most developers reach for, but few actually come near.

As business matters pertaining to projects that aren't Defense Grid 2 are outside the scope of this reporting, I'm not privy to the details of Hidden Path's finances, what potential games Hidden Path might be working on next or who the company is "biz deving" with. And in cases where I am made aware of that information, as part of my agreement with Hidden Path (and out of respect for the sensitivity of their business matters and potential impact on employees and partners) I do not report it.

What is safe to say, however, is that Hidden Path as a whole is in the same boat as Lex Story: wrapping up the tasks on its plate and looking ahead to whatever will be next. And perhaps worrying there might be too long a gap between one and the other.

As for all of the milestones staring the company in the face for Defense Grid 2, the team will hit them all. Daud will be in the office until 11 p.m., making the game in his spare time. But the co-op level will be ready for E3. The text will be ready for localization. The scripts will be ready for voice recording in Los Angeles. The publisher milestone will be hit. The clock will be reset, new milestones will be added to the calendar and the development of the game will continue.

There are mere weeks left on the schedule before the console version of Defense Grid 2 must ship for certification, and slightly more than that left before launch. This is game development.

"'We'll get close enough to the end where we'll just drop it on him like a bomb.'"

- Addendum -

Just days before this article was set to publish, Hidden Path informed me that there have been changes to its team.

The "biz dev" has been successful, to a point. Starting immediately, Hidden Path will begin work on a new project, which will sustain the studio and much of the Defense Grid 2 team as it rolls off the project.

In spite of it coming at a strategically opportune time for Hidden Path, however, it will not be a project that requires certain artistic specialities, such as those of two employees let go. Those two employees are Kevin "Particle Man" Loza and Defense Grid 2 Art Lead Lex Story.

According to Hidden Path, there wasn't a place on the new game for artists with their unique skill sets, and there was no extra room in the budget to keep them around while the company found other work for them to do.

Story took the news hard, but he doesn't blame anyone. He recognizes he's a very specific type of artist, with a mechanical engineering background and a penchant for creating weapons of war.

"I'm not gonna let it crush me," Story says. "I'm not a delicate flower by any means."

Story is currently deciding what his next project will be, or if he'll even remain in the game industry. His first game was Return to Zork. He's a 21-year veteran of the industry and a Marine. And a trained chef. He is intensely aware that his background and experience make him both extremely capable and also potentially harder to fit in a variety of teams.

This is not unique to Story.

The video game industry changes so rapidly its most accomplished veterans are often evolved out of the ranks, or become too expensive to keep on the team. Although Story's situation is more the latter than the former, he is nevertheless now on his own, looking for a new place to call home.

And this, all too often, for far too many game developers, is also game development. Babykayak

Continue to part nine of this series here.

This story is part of a series covering the development of Defense Grid 2. To read previous installments, please visit the Making of Defense Grid 2 page. This series will continue into mid-2014, the projected launch date for Defense Grid 2.

Images: Hidden Path Entertainment, Polygon