To delay or to crunch: The making of Defense Grid 2, part four
Tearing it down. Starting over.
It's a Monday in December 2013 and I'm in Bellevue, Wash. to visit Hidden Path Entertainment. Since May I've been following the development of Defense Grid 2 from start to finish, interviewing every team member, reading every design document.
On my first visit to Hidden Path, Defense Grid 2 didn't exist as a game. It was just some concept art and programming code that didn't do anything yet. By my second visit, the team was working on a "first playable" level to show to investors and take to the Penny Arcade Expo to show to the public. The game functioned, but it was definitely still a work in progress. My third visit began before PAX started and ended during it.
The result of this ongoing experiment, so far, has been an unprecedented look inside the process of creating a modern video game. Hidden Path has left every door open. No meeting has been off limits. No team member "unavailable." And slowly, over the course of seven months, a clear picture has emerged of exactly how a video game is made and why that process is fraught with complexity.
It's been three months since my last visit, since the independent studio's unveiling of DG2 at PAX when it showed the game to the world for the first time.
The struggle in September was to simply get the game functioning well enough to show it off. It showed one level with one game mode — a multiplayer battle level called DG Fighter — intended as a representation of the final game. The team's hope was that players of the original Defense Grid would find DG2 on the show floor, play it and enjoy it, then get excited for the game's eventual beta and then final release. It worked, to a point.
People who played DG2 at PAX enjoyed it, almost without exception. So far so good. As for as people finding it — not so much.
A minor delay here, plus a minor delay there, plus a minor delay somewhere else equals months of planning turning on a dime.
"The sign [at PAX] only said DG2," says Executive Producer Jeff Pobst. "A lot of people knew [that meant Defense Grid 2] and were good, and a lot of people were like, 'What's this game? Oh, this is Defense Grid?' But they didn't get that from the sign."
That problem called for a branding change. DG2, when it launches in 2014, will be called Defense Grid 2. As for precisely when it will launch, that too may change.
The Defense Grid 2 team is now in factory mode, putting pieces together in stages, building game levels one at a time, chugging away toward a finished game sometime in 2014.
The level it showed at PAX was almost completely scrapped after the show. The temporary art and code was re-engineered, so now it is radically different than when I saw it last. It is also, due in part to planning, but also to unforeseeable setbacks, the most complete level in the game. Still and again.
Changes to the game engine have slowed the progress of building new levels. Art changes required the tedious reorganization of work already finished. And the team's plan for offering user-created levels through Valve's Steam portal has been slow to come together, due in part to the continuing evolution of Valve's own development on Steam.
In the three months since finishing one level of Defense Grid 2 to show to the world, the team has mostly completed just one additional level. That's two out of 20. The remaining 18 are still works in progress, and the only working level they can show me is the same working level I saw for the first time more than three months ago.
The original schedule called for the beta release to begin in December, just a few days from my arrival. I'm here specifically to see that begin, in fact, but it's not going to happen. The beta is being rescheduled. The launch is TBD.
A minor delay here, plus a minor delay there, plus a minor delay somewhere else equals months of planning turning on a dime.
This is game development.
The decision
The decision was made in the fall.
Four people sat in the room with the giant TV on one wall and the illustration of a placid forest scene on another — the Hidden Path conference room. The room can hold two dozen or so people, seated in leather chairs around a polished wood table. On this day, the room is almost empty with only four people. But they're four pretty important people.
Producer Dacey Willoughby, executive producer Jeff Pobst, lead designer John Daud and PR/marketing director Shannon Gerritzen are here to discuss the future of Defense Grid 2. It's not an emotional meeting, but it's a meaningful one. The game is running late. The question on the table: delay the beta or work overtime during the holidays just to hit that mark.
This is the side of game development you never see: where the decisions are made that will impact the rest of the team for the rest of development. (Full disclosure: I didn't see it either. This meeting occurred in my absence. I was briefed on it via Skype immediately after, and again in person during my next visit.)
John Daud will design hundreds of levels for every one that makes it into the game.
These are the challenges of game development at its core: Art is being drawn. Code is being programmed. Levels are being designed and sound is being recorded. All the work of the disparate creative disciplines plus the energetic whiz bang of technology creation are being conducted, simultaneously, to eventually be brought together at a specific date, in a specific order, to be assembled into a game. It is a symphony of elements that, in and of themselves, are hard to manage with precision and must all be brought together in harmony, on a budget and on a deadline.
In the case of Defense Grid 2, it's not coming together as expected. But here's the dirtiest secret of game development: It almost never does.
it's not coming together as expected. But here's the dirtiest secret of game development: It almost never does.
The majority of the Defense Grid 2 delays stem from what should be (and actually is) making life simpler for the team, the Hidden Worlds engine. Developed a decade ago for a game that was never completed, Hidden Path is now reviving the Hidden Worlds engine for use on all of its games. Which, for now, are DG2, the open world construction game Windborne and another game I've been informed of but haven't actually seen (and absolutely can't write about). The idea is to make Hidden Worlds a platform for any type of game HPE might want to create in order to make it modular and reliable, and to do the work of building it — most of it anyway — once, so that the company can then turn its attention to making fun games, rather than to continually building technology.
The plan is a good one, and it's not new to HPE. This desire to make game development simpler is part of why engines like Unreal and Unity have enjoyed so much success. Building technology is often a distraction from making games. And it can be a never-ending process.
The evolution of game technology often moves faster than game development. So a large percentage of the time spent making any game with a custom engine is spent working on that engine so that the game itself will look as good as its competitors' games. This is why, in a nutshell, building Duke Nukem Forever took more than a decade.
And so, Hidden Worlds. For the most part, the engine has worked just fine. It was advanced for its time a decade ago, and the various teams at HPE have been updating it as needed. And that's why DG2 is running late.
And that's why they're here, these four, in the big room with the big TV — to talk about delaying the game. For Gerritzen, the decision is fraught. Months of scheduling and precise planning have been built up around the 2013 holiday season. She's paved the way, through marketing with the public and direct outreach to the press, for the beta to be well received before the end of the year. Delaying will take some of the edge off her work, and may require some rebuilding of trust.
But for Pobst and Willoughby, the decision is an easy one: The game isn't ready, and Hidden Path doesn't crunch. Not, at least, unless it is vitally necessary.
Hitting beta on schedule would be ideal, but it's not critical. The beta will be delayed. The marketing will just have to deal.
Instead of crunching to hit the beta, the team will continue on. First order of business: tearing the PAX level apart and rebuilding it.
"I try and identify what choices we're making, even when we don't know that we're making choices," says Jeff Pobst (left).
After PAX
"In my mind, our PAX build was almost like a proof of concept," says producer Dacey Willoughby. "We put things together so we could say, 'This is how you can play the game.' ... Following PAX, you have to go in and evaluate what you have left, what things you have to change from that build.
"So basically, if you're looking at it from the outside, we had this build that was beautiful, and then we took it and just broke everything."
Showing off Defense Grid 2 at PAX was a major milestone for Hidden Path. But it was also, in many ways, a distraction. The opportunity to gauge player interest and get feedback while development is still underway was invaluable. But orienting the entire team to provide a complete, playable experience before any of the game's major elements were finished was a potentially disastrous challenge. Balancing those two halves of production fell, largely, on Willoughby's shoulders.
"When we do company meetings here, we like to be able to show the build off during the meetings," Willoughby says.
We're in the conference room with the big TV. Willoughby is mousing around in the completed level — the PAX level. She's catching me up on the past three months of breaking the PAX level and then rebuilding the game around it.
At a glance, the level looks radically different. Where there was once water, there's now lava. Where the aliens — the mobs — once looked like slightly humanoid aliens, they now look like a completely different type of creature. And there are more of them. Where towers once looked more or less modular, they are now various shapes, colors and styles. And there are also more of those, too. Everything has received a dramatic reworking, artistically. More elements have been put in place and the underlying game engine is much more robust.
The result is much more interesting — and much more playable — than at PAX.
The result is much more interesting — and much more playable — than at PAX. This is the level as it will most likely ship with the game — the level as it could have been completed long before now, had it not been required to be completed, in a less finished state, in September for PAX. The before and after highlights the painful process of balancing what must be done to produce and market and sell a game versus what could be done if nothing mattered but the production. And this dichotomy, in some part, is why the producers voted to veto the push to hit the marketing deadline for beta.
Willoughby says that at any given time, just to show off the game in its current state takes about half a day of dedicated preparation and building. Even to just show the game to other people within the studio. Or to me.
"It's not that we don't like showing the build," Willoughby says. "It's having ... to make special arrangements to get something like that up and running. ... It's a little frustrating. It's not a huge deal, but it's kind of like, 'Oh, we have to do this?'"
A dog and pony show.
just to show off the game in its current state takes about half a day of dedicated preparation and building.
"Yeah," Willoughby says, smiling. "I didn't want to use that term, but ... You know. Those types of things can be pesky, I guess, is a good word."
Since PAX, Willoughby's task list has included, among other things, helping the art team to revamp almost every visual element in the game. An effort lead artist Lex Story describes as "kind of crazytown for a bit."
"The look of the towers had changed, because we were having problems at distance, differentiating between gun and cannon," Story says.
Story's obsession is to bring the towers back to scale, and it's an understandable one for a game that players engage with from a 10,000 foot view. At that height, the gun towers, canon towers and laser towers all looked the same. Massive weapons of war, certainly, but all extremely similar. A redesign was necessary to make each look more unique, so that players could differentiate them in the middle of a tense match. Same with the mobs, which looked fine, but not fine enough. They weren't "there," according to Story. Those got reworked, too.
And then came the textures. Every object in a game is constructed of polygons drawn by computer code. Those polygons are then covered over with textures, "materials" drawn by an artist and attached to the coded polygons using even more code. Textures are what make a polygon look like a rock, or a tower.
An update to the Hidden Worlds engine required the recoding of every material and the art textures that make up those materials. Each material file — thousands of objects — had to be re-attached to every polygon in the game by hand, one at a time. The kind of mind-numbing work that's less art than data entry, but it had to be done. The engine had to be tweaked, and so the art had to be tweaked. It's the kind of small thing that can turn into a big thing in game development.
"We had known we did something kind of temporary for PAX, and then we went and did the real stuff," Pobst tells me.
We're in his office with the rocket scientist knickknacks and the glass wall looking out on the development floor. We're interrupted occasionally by someone checking in or needing a quick answer. And Pobst is somewhat distracted with the swag giveaway he's overseeing later that week.
Being in the game business, Pobst and other high-level team members frequently get gifted t-shirts and other branded gewgaws from game development neighbors or potential business partners. Pobst tries to give this stuff out to other members of the team each year. There's a drawing. People take turns. Later this week a ping pong table will be wheeled out of storage and covered with a mountain of swag. By the end of that day it will all be gone, squirreled into backpacks and cars, dispersed to the various homes of the members of Hidden Path.
When the decision came to build entirely new engines for Hidden Path's new games or to refit Hidden Worlds, it was to the people at Hidden Path that Pobst and Chief Technology Officer Michael Austin turned. Many of the people who had created the Hidden Worlds engine a decade ago still work at HPE, including Pobst and Austin themselves. The hope was that the company had retained the knowledge of what went into that engine, and could apply it to refitting the engine in progress as well as integrating it with the new games.
Rather than spend a dedicated amount of time examining the engine specifically to relearn what it contained (and didn't), the HPE founders decided to press it into service immediately and deal with what problems may come as they came. Months later, as those problems have now begun to cause delays to Defense Grid 2, Pobst acknowledges this may not have been the better plan.
"We could have done a more thorough investigation of the code base ... to understand our task base better," says Pobst. "I don't know that we would have saved any development time, but we would have known sooner if we had done that. And then the question is, was knowing sooner worth being a week behind for PAX?"
These are the kinds of decisions that plague Pobst, as executive producer on the game and CEO of the company. What decisions will impact what future decisions, and which will simply bog the team down, spinning its wheels on problems not worth solving. And whether or not a week taken here will save two weeks down the line, or vice versa.
"I try and identify what choices we're making, even when we don't know that we're making choices," Pobst says. "Sometimes it's OK. You're going to pick a thing. You're going to decide, yeah, that's what we want to do. But what's more dangerous is when you've made a choice and you didn't know you made it."
The decision to not stop and examine Hidden Worlds before deploying it may have been one of the latter, a decision that subsequently made other decisions the team didn't even know were on the table.
The alternative is where Defense Grid 2 is now: It was on time for PAX, but it is running behind for beta and potentially for launch.
The PAX build served one main purpose: Show off the game. And it served that purpose exceedingly well. But it was a bit of theater. In the controlled environment of PAX, it gave players a sample of the experience of playing the finished game. Outside of that controlled environment, it completely fell apart.
Outside of that controlled environment, it completely fell apart.
In the three months since PAX, Hidden Path has been rebuilding that experience, but this time for real. Now it's an actual working level that will form the core of the finished game.
Now all it needs is more levels.
Factory mode
It's Tuesday morning. 10 a.m. — early for most game developers.
I'm sitting in the development area for Defense Grid 2. A few desks have been removed. What, on my last visit, was a crowded and humming space is now somewhat more spacious. Gone is the new row of desks and the line of Ethernet cables taped to the floor.
A company-wide "360" performance evaluation returned the verdict that this area was too cramped. People wanted more space. So out came the desks. Some team members were moved to areas nearby, while others were transferred completely. Among them is programmer Peter Freese, who has shifted over to work on the "core tools" team performing maintenance on Hidden Worlds.
Sitting in his usual place by the window, walled off by cubical partitions, is John Daud. Weak Seattle sun streams in from the glass behind him. Beside him is his ever-present notepad, containing notes from meetings with the team and Pobst, tabbed with scraps of blue Post-its at places he wants to refer back to. Underneath the current pad are the ghosts of notepads past, full pads that, in total, tell the long history of Defense Grid 2's development to date.
Daud moves his knee and shows me a drawer full of other such pads from other games. Enough to fill several boxes. If one were so inclined, one could take those pads and, with a little luck and the ability to make the thousands of decisions Daud made over the course of dozens of years, exactly the same as he made them, at exactly the right times, one could recreate his games. In other words, there's small chance of that.
What were once vital tools for creating video games are now just old notepads taking up space. Perhaps they remind Daud of all he has accomplished to date and bolster his confidence as he does it again with Defense Grid 2. The notepad at his right hand will some day join the collection. But until then, he has a lot of work to do with it.
In front of Daud are two screens, each presenting a level in progress. On one screen is the near-final level with complete art: full color, lots of texture. On the other screen is the same level in a "grey box," in the editor. There's no art, just shapes and lines. The grey box level is what he works with.
Daud builds the Defense Grid 2 levels from scratch using the editor created by Matt "Twig" Johnson and the programming team. It is more or less the same level editor that will ship with the game. Players who so choose will be able to build their own levels, using pre-made sections of road and art "kits" for different-looking worlds, and then offer those levels through a Steam storefront that rookie programmer Charlie Ngo is, at this moment, still creating.
Daud's levels will ship with the game. Daud's levels will, for all intents and purposes, be the game. To say that ensuring they are fun and inspiring stresses him out is an understatement. You can read the stress in the look on his face, hear it in the sound of his voice and measure it by the velocity of the objects he occasionally hurls when things go wrong.
Daud's "outbursts," as the teams calls them, are the stuff of company legend.
"I had two outbursts [in three months]," says Daud. "One, the editor was acting up, and I ... [he chuckles, remembering] ... Peter [Freese] assured me that whatever work I did would not be lost. I did something, and I lost my work. I may have thrown a Kleenex box at the wall behind [him]. [Freese] doesn't sit there anymore. I don't know why ... I feel great remorse."
Other team members within earshot laugh at the retelling of this story, but there's tension in it. Everyone understands. It's not the outbursts that are making them nervous; it's the pressure to complete the game.
Daud is at the front of the assembly line. So far he has built hundreds of levels. He spends his days switching between building more and polishing the ones he thinks might be (some day) good enough. He meets with Austin to talk theory and comes away with plans for even more. The levels in the game will progress in complexity to provide a challenge for players as they get better at playing. But it's the early levels that give Daud headaches. As the designer, his own knowledge is so advanced it's hard for him to create something that won't be too challenging for beginners.
"I've been doing this long enough that I start to think that, 'Oh, this is boring, this is not going to be interesting.' Problem is, players need to get acclimated to the game," Daud says. "Meeting with [Austin], we map out what the expectation is for that level. What's it teaching? What's it introducing? And then we build it accordingly."
What's also eating at him is the exact opposite of the beginner levels: the game's conclusion. Daud wants it to be "epic" and right now he's trying to decide just what that means. Will the final level be bigger? Will it have more layers? More mobs? Or will it be an experience that transcends the basic tower defense aspect and adds a new element, a new method of interaction? Something like "the key."
A week and half ago, during a brainstorming session for the artists, someone voiced the thought: "What happens when you move from level to level in the game?" This led to discussion of how the player could be brought physically into the game worlds, ferried (as if on a boat) from map to map.
"How does that work?" Story recalls asking. He's animated. He likes telling this story. Although he prefers calling the key "the boat." "Is it wireless transmission? I was like, 'You know what? What would make more sense is if you actually gave a reason you were accessing the resources that are built into the planet.' That's what the key does. The key ... comes down, it locks into one of the build squares, and then you see the whole grid come alive. Vrrrrrrrrm! It's just like you now have control! It's like, 'Oh, whoa!' And then at the end of the mission it folds up and takes off and launches. BSHHHHH!
"Hopefully with dancing shadows on the ground. The flicker and the wash of the ambient light from your exhaust."
The key is still a work in progress. It may become an integral part of the game, something that provides a gameplay benefit. Or it may remain an aesthetic enhancement. Or it may go away all together, replaced with some other idea or simply abandoned. This is game development, where anything can happen, and the final game is still many months away.
This is game development, where anything can happen.
"We'll see what happens," says Daud. "[The Key is] built and animated. It looks cool. It'll be in the game one way or the other."
It was brainstormed, built and animated in a week and a half. While minor annoyances with the technology are slowing overall progress on the game, the weight of the creative talent being brought to bear on Defense Grid 2 compensates by enhancing and adding flair. Everyone agrees that when it is finally complete the game will be much more advanced, ambitious and interesting that what was originally conceived.
For now, though, they're building the levels, one at a time, marching steadily toward the end date.
"I will probably start panicking this month," Daud says, laughing. And as he says it you can see the panic has already begun.
as he says it you can see the panic has already begun.
Getting it right
It's Tuesday afternoon. The core design team is deep in discussion about shields and lasers.
"John wants you to see the laser going through, and the stuff going on inside," says Jeff Pobst. "Other towers, you don't have an item for shooting the shield and they're doing some damage to the shield. Before the guy dies, the shield might go away. That's all I'm saying."
Pobst is sitting in a swivel chair in the middle of the room in thoughtful repose, leaning back, wrists crossed behind his head.
Standing around him are John Daud and Lex Story. Effects designer Kevin Loza sits next to him. Willoughby is at her desk, listening, taking notes.
As the elements of the game come together, unanticipated events often result. In this case the team has discovered that different weapons do different damage to a mob depending on whether they are hitting the mob itself or the mob's shield. And a planned in-game item that would allow certain towers to ignore shields entirely is causing problems in how the game displays that damage. Specifically, multiple towers could be doing damage to a mob's shield; meanwhile another is simply ignoring them and doing damage to the mob itself. If the mob dies before the shield fails, it's obvious what happened. But if the mob is killed at the same time (or after) the shield fails, that could send the player the wrong message.
The issue was noticed, a conversation was begun and that conversation snowballed into an impromptu meeting. This happens often enough to be called normal.
Story suggests a color change to indicate the shield is being affected. Pobst loves the idea of a color change. It's "nice and cheap." Meaning it won't affect the game's performance and can be implemented without a lot of fuss. But it won't solve the current problem.
"If the beam hits the shield, the whole shield changes color," says Pobst. "That actually implies the opposite, though, that we're doing something to the shield. As opposed to the lasers ignoring the shield. But if the laser hits [the shield] and a spot in the shield, ideally, goes away ... It's just not there anymore. That would be really more interesting. Or it went to a lighter color or blue or whatever."
Story: "What I would like to see is the actual obviousness or the intensity of the shield becoming more transparent. And then it has a very large effect of showing; that's where it's impacting."
Pobst: "It's almost the other way around. Where the laser is going through, that's where you want it to be most transparent, to imply that the laser is unaffected by the shield."
Story: "I see. I'm using a different logic."
Pobst: "Yeah, I know."
There is laughter all around, then the conversation refocuses and a long debate over the relative merits of color changes versus graphical effects ensues. By the time the game is finished a variation of this conversation will have taken place over every single detail. Thousands of decisions about millions of details, major and minor, that will all coalesce into the final product. Tabbed in Daud's notebooks, then put in a drawer.
"It's not perfectionism. It's just getting it right," says Story.
"It's not perfectionism. It's just getting it right."
There is laughter, and the suggestion that this might make a good quote. And it does. The obsession over details that many players will never notice might seem, to some, like insanity. But this is game development. Each small decision influences the next, and the end result, if they get it right, is a whole that "feels right" even if you can't explain why.
Two levels are finished, and there's 18 to go. 
The story continues in Part five.
This story is part of a series covering the development of Defense Grid 2. To read previous installments, please visit the Making of Defense Grid 2 page. This series will continue into mid-2014, the projected launch date for Defense Grid 2.
Active Discussions
Polygon Quarterly: Not for the Shareholders but for the Games
in Off-topic by sum_guy
Polygon forums close this Friday
in Off-topic by Chris Plante
Looking for more people to write Plight!
in Off-topic by Prinny
Welcome to the >new< forums!
in Off-topic by Prinny
What are you reading manga right now?
in Off-topic by lee87645312