Making Gangnam

How Harmonix turns music into fun

Jump to

You wouldn't know from looking at it that Harmonix is a game company.

From the outside of its squat brick and glass office building (one of two the company occupies in Cambridge's Central Square) only the painted "Harmonix" logo gives any indication as to what's inside. (The building's other occupant, Harvard University, doesn't even have a sign.) And even that is so small you might miss it.

Inside, it's pretty much the same. There's no fancy architecture, no foosball tables, no massive kitchens filled with free soda. No climbing walls or roller coasters. Only a long wall filled with trophies and console games gives any hint as to its business. Otherwise, it's cubicles, conference rooms and hallways filled with junk. It could be any office anywhere. One whole room, containing a suite of top-of-the-line motion capture technology, is practically walled off by a pile of discarded office chairs and computers and banks of dusty routers vying for floor space with pallets full of The Beatles: Rock Band, still in cardboard shipping boxes.

Harmonix, by all appearances, is the anti-game company. And as far as the people who work there are concerned, that's exactly how it should be. When asked why Harmonix hasn't taken the time to install fancy lighting or life-sized statues of game characters, one publicist says, "We're too busy making games."

He says he knew that a VIP was en route for a visit when a pile of discarded computer equipment suddenly vanished from the hallway outside his office. The VIP? Usher, whose personal choreographer also worked on Dance Central 3.

This deep connection to music and how it is made is part of the secret sauce of Harmonix. The name, in fact, says everything. It's not Harmonix Studios or Harmonix Games — it's Harmonix Music Systems. As one producer says, it could be a speaker store. And in a way, it is.

Harmonix creates technology that allows people to enjoy music in new and different ways. Only it doesn't deal in woofers and tweeters, they deal in games. Digital media that allows listeners to step into the experience and become part of the song.

Its latest, <em>Dance Central 3</em>, uses the Xbox Kinect technology to allow gamers to dance, controller-free, learning dance routines step-by-step with the help of the in-game characters and dance coach Usher. The game offers over 200 dance hits and counting, one of which is exactly why we are here.

Harmonix has invited Polygon to spend a day in its music lab, observing the process of translating a hit song into an interactive dance number. The song: "Gangnam Style," the K-pop hit by South Korean artist PSY.

On the day we arrive in Cambridge to begin our tour, Harmonix has just announced the release date for this add-on song, but the process of transforming it into a game experience has been going on for weeks, and is, in fact, still ongoing. What follows is an exclusive look at a process that has helped <em>Dance Central</em> to become one of the highest-rated music games: a unique blend of creative vision and technological design that has propelled Harmonix Music Systems from a quirky experimental game company to one of the most trusted names in music games.

The Capture Lab


During our time at Harmonix, we're told repeatedly that there are things that can't be shown to us. Certain rooms. A whiteboard. Entire buildings. As at any game development studio, there are people working on projects that haven't yet been announced and may never see the light of day. Those things and those people are understandably off-limits.

Yet even within the confines of our tour of what goes into making Dance Central 3, there are doors that remain closed. What's behind those doors? Officially: magic. Unofficially: technology that's so far advanced it may as well be magic.

In a game like Dance Central 3, you, as a player, are standing in front of your television and dancing. The game can see you, thanks to the Kinect hardware, and it can tell how well (or not) you are performing. It watches you succeed and fail. It judges you. And then it makes a decision about how well you did and gives you a score.

Make no mistake, whether or not this is actually magic, it is magic. A century ago, seeing such a machine in action would cause panic. Today, when almost every household device is wired to the internet with a camera and a microphone, it just seems like another one of those things. Like the refrigerator watching you making a snack. Big deal.

You may not understand precisely how Dance Central 3 is able to judge you; you just know that it does and that it has something to do with the cameras in the Kinect and the amount of light in your room and the space between you and your sofa. Maybe also humidity. For those who play Dance Central 3, all we need to know is that it works. For those who make it, it's a whole different story.

Harmonix Music Systems is literally right next door to MIT, the place where many Harmonix employees earned their schooling. The word from these people is that a lot of what happens inside your Xbox while you are being judged by Dance Central is "magic." And when they say "magic," what they mean is "proprietary technology," which means that we don't get to see it.

Here's what we do get to see:

I'm standing in a room in a basement. Just outside the door is a pile of office junk so massive and precarious that I'm afraid it could topple at any moment. Inside the room, the paint is peeling. There is no chrome; no flashing lights. The only clue I'm about to see the first step in a process involving high technology and massive amounts of creativity is the man standing beside me holding a spandex suit.

The author with Adam Repka (right) in the Harmonix capture lab.

This room is the Harmonix motion capture lab, and the man beside me is Adam Repka, one of the animators at Harmonix who helps capture the movements of human dancers and translate those into video game form. The suit is one of Harmonix's custom motion capture, or "mo-cap" suits, fitted with wires, a battery pack and dozens of tiny LED emitters.

"There's a good amount of set-up involved to get somebody in the suit," Repka says. Most of that set-up involves correctly placing the emitters, which are precisely sequenced up and down both arms and legs and across the dancers' bodies. There are 40 of them, and they have to be attached precisely or the data they generate will be meaningless.

This is the first part of the "magic." It begins with the mundane: putting on clothes, attaching tiny sensors. On a nearby computer screen, Repka shows me a reconstruction of the data created by his LED sensors and captured by the 24 light-sensing cameras positioned around the room. On the computer, the data is represented by a stick figure made of tiny dots, with each dot corresponding to an LED sensor on the body of the dancer. When the dancer moves in the room, the stick figure in the computer moves with it. Magic.

"They go from A to H," says Repka, referring to the LED emitters and pointing to a diagram on the wall. "The way it works, it has strings of lights. One string goes up around the head, so you can see A-B-C-D. Another string goes around the torso, you can see A-B-C-D-E-F. Another string goes down each of the legs and down each of the arms. It's really flexible. This is all configurable. If you want more on an arm, or more on a leg, or less, or [if] you want something completely different, like a prop that only has three markers, it's totally configurable."

More sensors mean more precision, which is useful because this is the real world, and real dancers sometimes do crazy things. Like cross their hands and gallop, or get down on all fours and hump the ground.

"That's one of the biggest problems we have while we're shooting," Repka says. "These guys are doing crazy moves. We've mo-capped people doing backflips, headspins, you name it. We don't want to limit them.

"Lights will fly off as they're sweeping their arms or something. It's always the extremities which are the biggest problem. They'll do something ... and tear a light off. So one person will be watching that to make sure if anything shifted or anything got torn off."

The entire process involves three animators. One watches the dancer to make sure they perform the moves correctly. The other two man the computers, watching the data stream and looking for any anomalies.

With "Gangnam Style," there's really only one problem area. They call it "the cat cow."

"It tests the limits on just the one move," says animator Andy Bouchard. "The down-on-all-fours move. We've never done a move like that before, and our system is not really built for getting down on your hands and knees. To get that to work ... it was really the tricky part in the song."

The "cat cow" requires the dancer to get on hands and knees, thrust their hips and swing their head from side to side. It is but one of a handful of ridiculous moves in a dance inspired by playing cowboy and humping things, but throughout the day we will hear from almost everyone we talk to that in spite of how ridiculous it is, it has been hellish to recreate it in the game. A lot of magic has been thrown at solving the problem of the cat cow.

But we've gotten ahead of ourselves. The mo-cap room is not where this story begins at all. This story begins with the song, "Gangnam Style," and the man who brought that song home to Harmonix, Chris Rigopulos.


LightsLED mo-cap lights (above), lit for testing (below).

The music master

Chris Rigopulos brings the two worlds together.

Chris Rigopulos is a quiet man. He is tall, but would otherwise melt into a room.

As we're waiting for him to arrive for his interview, he pokes his head through the door, looks around, apologizes and excuses himself. Having never met the man, I assume he'd just gotten the wrong room. When he returns a few moments later, being gently led by his publicist, we both sheepishly smile, acknowledging our mutual mistake.

It is the kind of encounter that, combined with the man's temperate personality, might make you forget he practically eats million dollar deals for breakfast.

"I'm ultimately responsible for all of our music industry outreach, partnership development and bringing music into our games ... which is kind of important," Rigopulos says of his job. He is not being ironic. When he says that landing the music for a music game is "kind of" important, that's just what he feels. "Having a dance game without music would be ... I don't know. Awkward."

Rigopulos, brother of Harmonix CEO Alex Rigopulos, has been with Harmonix, off and on since 1997, when he worked on music licensing for Harmonix games that "nobody's ever heard of." Between then and now, he's worked in music licensing outside of games and consulted with major record labels. He returned to Harmonix full time in 2009, just before the company began buying itself back from MTV/Viacom's video game division.

"MTV had primary responsibility for music partnership development, so when we became independent again we had to pick that up and piece it back together," says Rigopulos.

Harmonix was still keeping up with weekly downloadable song add-ons for Rock Band and developing Dance Central 2. Losing the MTV connection could have dealt the company a hammer blow. It didn't. Credit Rigopulos.

"We invest a lot of time, attention, money, whatever into making sure that the games we put on the market are good," says Rigopulos. "Look at the Metacritic ratings on our games. Dance Central has set the bar and exceeded it in each iteration in terms of the quality. It's the highest-rated dance game by far. And not just that. Objectively, the Metacritic ratings are high 80s now. That's good by any standard, for any game. The same was true for Rock Band. Rock Band 3, our last Rock Band game with the instruments and all that, was a 93 Metacritic title.

"When you've got that kind of track record for quality, that's how you maintain the credibility. ... It makes it a little bit easier when we have that to fall back on, when we're trying to work with high-profile artists. Otherwise it would be a much tougher hill to climb."

Rigopulos doesn't consider himself an expert in promoting music artists, but he does understand that a 21st century music company has to be open to new ways of experiencing music and reaching potential fans. He believes that's where Harmonix comes in.

"There's a whole bunch of people that really care about video games," he says. "They care passionately about them. ... Harmonix is arguably the best pathway to reach that audience for musical artists who are looking for a way to gain some audience in that particular medium. That's where we fit into the ecosystem. It's a symbiosis in a way."

The artists make the hits, Harmonix makes the games and Chris Rigopulos brings the two of those worlds together.

But the song ... we're here for the song. Or in this case, The Song. The K-pop megahit that blew up YouTube and made millions of people want to giddy up like a South Korean cowboy. Let's talk about The Song.

Let's talk about "Gangnam Style."

The Song


According to Rigopulos, a hit as big as "Gangnam Style" comes with its own set of problems.

"Psy is everywhere now," he says. "It's unbelievable what he's accomplished, what he and his team have managed to do. The demand for his time and for his music and this song in particular is ... it's just astonishing.

"One of the biggest challenges for us in getting the license for the music and getting the license for the choreography was that ... everyone's trying to capture the lightning in a bottle."

According to everyone we speak with at Harmonix, this is the key: it's not just about the song, it's also the choreography. For a song that is much about the goofy spectacle of Psy doing his dance as it is about his music, you can't have one without the other. Harmonix wasn't even willing to consider releasing the song for Dance Central if it couldn't also acquire the rights to the choreography.

"We were getting a couple of people tweeting at the Dance Central account saying, 'Hey, have you heard this song? This is really cool! I think it's going to be really big!'" says Annette Gonzalez, community manager for the Dance Central games. "I had seen a couple tweets about it and sent it around. Slowly, as the months went on, we kept getting more and more requests. We realized this was going to be really big, and we needed to get on it now."

Dance Central 3 had just been finished when "Gangnam Style" exploded all over the internet. The song list was done, and the initial batch of DLC songs had been locked down. Harmonix could have (maybe should have) let it pass, but to do so would have been to ignore one of the defining dance sensations of the decade. It simply had to find a way to get it in the game.

The result: weeks later, the team is still crunching on just the one song, long after the release of the game and a week before the Thanksgiving holiday. The office is almost empty save for a skeleton team working on "Gangnam Style." The release date has just been announced (it is barely a week away), and the team appears energized, if exhausted.

"Everyone at Harmonix has such a huge passion for music," says Gonzalez. "It makes a big difference in the work that we do. It's a labor of love, a little bit. We put a lot of work into it."

We're standing outside the Harmonix office building on a brisk, sunny Massachusetts morning and Gonzalez can't get her phone to stop buzzing. She was responsible for announcing the release date just a few moments ago and responses are pouring in.

"I think the popularity of 'Gangnam Style' ... fits right in to the weird sort of viral YouTube internet wormhole that you fall into at three in the morning when you see something really crazy," she says. "You're like, 'That's weird, but I really like it!' It's a super catchy song. You don't really understand any of the lyrics, but you can imagine it's probably something pretty cool. Then the dance and the video are just so ridiculous. You can't help but share it with your friends."

For Rigopulos, Gangnam Style was a no-brainer. The kind of song he'll chase all day, because it exemplifies the spirit of fun and community Harmonix is trying to engender with its music games.

Gonzalez doesn't just promote Dance Central, she also plays it. Mostly to get better at dance moves so she can show off at the club.

"I always wanted to be a dancer," she says, "and I never got to do that. I was too afraid to take classes. Because of a game like Dance Central, I've been more comfortable dancing at shows and dancing on stages and doing different sorts of performances. I started taking dance classes. It opened me up to a lot of stuff, and I don't know where else I could work that would offer me those kinds of opportunities. I'm really grateful to be here.

"I always feel like a rock star. There are some places in Boston that I'll go to that actually tend to play most of our soundtrack, because we have a lot of hit songs. ... I got my roommates to start playing Dance Central, so I got them to start doing the routines at the club, and they think we're a dance troupe. It's pretty hilarious, but it's a good time."

For Rigopulos, "Gangnam Style" was a no-brainer. The kind of song he'll chase all day, because it exemplifies the spirit of fun and community Harmonix is trying to engender with its music games.

"In the case of 'Gangnam Style' in particular, it's this phenomenon that just kind of happened," Rigopulos says. "It didn't even just happen for us. Our interest wasn't like, 'This is a marketing opportunity.' One of our resident K-pop experts forwarded it around, the YouTube video. Not because it was a pitch to go and license the song, but just because it was fun. ... Then it started snowballing."


The rock star

For a company with so much experience with actual rock stars, Harmonix is surprisingly free of rock star game developers. Then there's Senior Producer Naoko Takamoto. She's the closest thing to a swaggering diva in the entire building, yet even on her, the act seems so natural you imagine she was born rocking massive dark shades.

"I'm from here," Takamoto says tartly, when asked where she's from. "If you're asking me what kind of Asian I am, I'm Japanese."

Once a song gets on the Dance Central roster, Takamoto takes over, coordinating between the creative artists who do the choreography and the technological wizards who turn the performances into a game. Small of stature, Takamoto moves like a dancer herself (she was a go-go dancer before working at Harmonix), knows her way around a recording studio (she also used to be a musician) and has the patience of someone with a lot on her plate and very little time for bullshit.

"I stopped drinking coffee," she tells me. "I switched to espresso because I don't have the time for the water."

Takamoto describes a typical day in her life:

"I go through e-mails and figure out what is the crisis of the day. It depends. When we're making songs, it's kind of ... I have nine people, nine choreographers, an authoring team, and QA. We're always plate-spinning a whole bunch of songs at once."

On this day, however, there's just one song: "Gangnam Style."

Harmonix acquired the rights to use the song fairly easily, but the choreography was the sticking point. That came along much later. Meanwhile the entire team had been waiting, desperately, to get to work.

"['Gangnam Style' is] a thing about people having fun with dancing and music. There's no shame in this song," Takamoto says. "It's just like, 'Yep, he's dressed up and he looks fly and he's running around and humping things and dancing and screaming.' You don't get to see that a lot. People just having all-out fun. It's nice to see that. It's refreshing. People want to jump in and be a part of that. The Macarena was like that, you know? The Humpty Dance was like that. There's no being too cool for that. If you think that, you just don't like fun."

The audience wanted "Gangnam Style" in Dance Central 3. Harmonix wanted Gangnam Style in Dance Central 3. Rigopulos paved the way, and then it was up to Takamoto. Once the rights went through, it was a mad dash to pick up and run the song through the complicated process of capture, animation, QA and "magic." The deadline was short and the pressure was intense, but for Takamoto it was just another day. She was born for this.

Then there's Senior Producer Naoko Takamoto. You imagine she was born rocking massive dark shades.

"Everyone wants to think they have the same idea of what the game is going to be in the beginning, but everybody else has a different idea of who they're making things for," she says. "Being a musician helped [me] because I'm used to having lots of different clients. You try to understand what everybody needs."

She says the art of producing a game like Dance Cental is trying to figure out how all of the different members of the team can combine their different creative energies to make one thing, and then making sure everyone stays on task and focused.

"Things are always kind of tense with video games," she says. "It's nice when you just have a stupid crazy deadline, because it's totally okay for everybody to freak out and be like, 'It has to be done now!' And you can get all your crazy out."

Takamoto was a starving musician when she was discovered by Harmonix. The company was working on one of its earliest hits, a rhythm game called Amplitude, and it needed someone with experience singing and writing songs. Takamoto hired on to write one song, then turned around and begged for a full time job.

She got lucky. Harmonix's next game was Karaoke Revolution. She hired on to "come in and sing all day," and never looked back.

"This place saved me," she says. "I have a grown-up job, and Usher's choreographer thinks I have the coolest job ever. If anything, I have accomplished being envied by Usher's choreographer, which is just ... that's headstone material right there."

The animator

After the song rights have been negotiated, the choreographers have been hired, the dancing has happened and been captured and the captured data has been analyzed, checked and verified, that's when Andy Bouchard steps in. As lead animator, it's Bouchard's job to make sure all of the other animators have the data they need and are on schedule turning that data into something someone can actually have fun with.

"The great thing about having our own motion capture studio," says Bouchard, "is that literally five minutes after we're done downstairs, we can come right upstairs and start putting it into Motion Builder and start animating."

Bouchard and the animation team have separate "skeletons" for each of the 10 Dance Central choreographers. Each skeleton roughly corresponds to the general height, weight and body style of its choreographer, which then corresponds to the game characters designed around those bodies.

After receiving the data from the mo-cap studio, an animator will simply align that data with the corresponding skeleton and literally "snap" data points to body parts. The result is an almost instantaneous animation — with some rough edges. Most of what Bouchard and crew do on a daily basis is sand away at those rough edges.

"If the choreographer, the guy that danced this originally, is dancing to it in the game after the fact and he's not seeing anything wrong with it, then that's awesome," says Bouchard. "Seeing people play a routine ... they're like, 'Wow, that choreography was awesome.' All that work, obviously, was done by the choreographer, but in the background I'm like, 'Yeah, it looks awesome because we spent a lot of time making sure that it was as close to the routine as possible.'"

For "Gangnam Style," Bouchard is hands-on. He's using the "Devin" skeleton, which corresponds to Devin Woolridge, the choreographer who re-created Psy's iconic dance moves and choreographed them into a Dance Central routine, then performed that routine downstairs in the Harmonix mo-cap studio, lit up like an infrared Christmas tree.


"If the choreographer, the guy that danced this originally, is dancing to it in the game after the fact and he's not seeing anything wrong with it, then that's awesome."

Bouchard lines up the capture data with the Devin skeleton and makes sure that, as the routine plays out, the data points representing the moves Devin performed in mo-cap correspond to how the Devin skeleton is supposed to move in the game and the steps Devin created for the dance. If everything works, it's a short day. If not, Bouchard has a lot of work to do.

"I'm going through, basically, almost frame by frame with the animation," he says. "This is Devin dancing here. I'm taking a look at what he's doing and making sure that my hand poses are matching his, so it's accurate. I'm making sure that things don't look broken. When this leg comes in, I want to make sure that knee is pointed in the right direction. It doesn't look totally broken. A lot of gamers can pick that stuff out. I want to make sure that stuff is all looking good."

Typically the Dance Central animation process will take two weeks per song. For "Gangnam Style," Bouchard and the animators were given one week. Then they were asked if they could do it even faster.

So far it's going smoothly. Bouchard is making minor fixes and planning to go back and animate the hands, which are not captured in the studio. But even there, he's caught a break: "Gangnam Style" doesn't have a lot of hand and finger moves. It's all "pointing, lassos and fists" which makes Bouchard happy.

Towards the end of our interview, however, something goes wrong. Another team member comes in, looking worried. The mood becomes tense. He and Bouchard begin talking in hushed tones. We are almost kicked out of the room, but Harmonix has promised us full access and it keeps its word.

What Bouchard has been showing us are theoretical flaws in the animation. Mainly just to demonstrate how the process works, but also because, until this moment, the animation process for "Gangnam Style" has been a smooth one. Now that's changed. Now there is an actual problem with how the dance has been animated, and the already strained production schedule could potentially fly right off the rails, less than a week before the DLC needs to go out the door.

If the problem is something Bouchard can fix, he'll work overtime to fix it. If not, it means calling choreographer Devin back into the studio to do another round of mo-cap. Which means more set-up, more QA, more animating ... and more time.

We are almost kicked out of the room, but Harmonix has promised us full access and it keeps its word.

Worst case: a shipping delay. If the dance can't be released as planned because of an animation flaw, that's a bad day for Bouchard. At the moment, it looks like whatever the problem is, it's big. Bouchard starts talking about calling Devin and scheduling more mo-cap.

"Maybe we do need a pickup with Devin," he says. "If I get the song done in three days, now I have two extra days to grab that pickup and I'm still not really behind. That's the point of rushing and trying to get it done as fast as possible. Everyone's on a tight schedule. If somebody can do their job just a little bit quicker, that might take some pressure off other departments down the road.

"I just hope Devin nails this ... otherwise it'll be really hard to hit that one-week mark."

The magic men

We're sitting in a rarely-used conference room tucked into a corner. Like the rest of the building, this room is drab to the point of dreariness. The closest it has come to an interior designer looks to have been some time in the late 70s or early 80s.

The walls, floor and ceilings are different shades of faded blue. One whole window pane is missing from what should be a stunning view of Cambridge and, just across the Charles, Boston. The windowpane has been replaced with a slab of plywood.

I'm told that before the repair, the glass had a gigantic crack in it and would shake and rattle in high wind. The room was being used by programmers at the time — who shrugged off the disturbance. Now the programmers have been shoed out, and we're using it as an interview space. Briefly. We'll be shooed out in turn moments into our interview, to accommodate a rare meeting.

What's remarkable about these mundane details is that they exist. Normally in dealing with the press, game studios will plan every encounter months in advance and adhere to a strict schedule. Money is spent specifically to ensure the absence of wrinkles and that no one ever sees anything so normal as a dreary conference room or a broken window.

At Harmonix, they are the honey badger — they don't give a shit.

In the dreary blue conference room, amidst the almost surreal aura of nonchalance, we're speaking to the two men who come closest to representing what we've been told all day is the "magic" of Harmonix. Nate Stoddard and Colin Sandel work, officially, in QA. They are part of the team that plays every dance, many, many times, helping to build what they call "filters" that translate the choreographed, captured and animated dance routines into something you can actually do at home and be judged on by your omniscient Kinect.

NateNate Stoddard

At Harmonix, they are the honey badger - they don't give a shit.

ColinColin Sandel

These men are not the final step in the process, nor are they the only team responsible for making the magic. But they're the closest we've been allowed to get to the process of juggling insanely complicated technology to make doing a horsey dance in your living room fun.

Stoddard is telling us the story of how "Gangnam Style" came to Dance Central 3:

"With any viral video, it's kind of like some major historical event, but it lasts a year or whatever," Stoddard says. "You hear a video or you see a page, and you think, 'Oh, I remember where I was when I first saw 'Gangnam Style.' 'Who sent you Gangnam Style?' 'Oh, I remember who sent me ...'

"It was the end of August and I was having a really bad day. A friend of mine sent me this random video to Gangnam Style, and then it exploded. As soon as it exploded, everyone was like, 'Oh my God, we have to get this song in the game.' I think our legal and our licensing department went overtime immediately."

Colin Sandel interrupts (He will do this often. The two men have a distinct "odd couple" vibe, earned, one assumes, from long hours spent completing each others' thoughts.):

"If I might interrupt," Sandel interrupts. "At the time, some of our community and production members ... were in Singapore, I want to say, and the song was popular throughout most of East Asia at the time. People asked them about it and they were really enthusiastic. Matt [Boch, project lead] said that he wanted the song in the game. At the time, I don't think we had anything in the works at all. It became a mad scramble at that point."

What Stoddard and Sandel do for a living can be summed up like this:

Harmonix uses some of the most advanced and expensive equipment in the world to capture highly detailed nuances of highly choreographed dance routines which you, the player, will then interact with using an astronomically less expensive piece of equipment in your living room. Negotiating the boundaries between the fine details of the mo-cap studio and the relatively less precise capabilities of the Kinect is where the magic happens, and Stoddard and Sandel are the masters of that magic.

I ask: If the trick is in figuring out how to translate the dances captured in their studio to dances you can perform in front of Kinect, then why don't they simply capture the dances with Kinect in the first place? Sandel practically chokes.

"We're using 3D models of the characters," he says. "While the Kinect builds a depth map of your body, the way that it actually tells where your limbs are is through the firmware. It estimates a skeleton, which is basically a stick figure, of you. Let's just say that if we tried to do it that way, our characters ... they would occupy a lower part of the Uncanny Valley, so to speak."

By which Sandel means they would look like ass.

Kinect, in short, makes a lot of "guesses" to compensate for the fact that its two cameras are essentially in one place, giving it a limited, almost 2D perspective. When Harmonix captures a dance, its using 24 cameras placed in a 360-degree array surrounding the choreographer, giving it a full 3D perspective. The trick, then, is to make it possible for Kinect to tell if you're doing what the dancer is doing, using less data.

Negotiating the boundaries between the fine details of the mo-cap studio and the relatively less precise capabilities of the Kinect is where the magic happens, and Stoddard and Sandel are the masters of that magic.

"If the grading was based on what the dancer was actually doing," Sandel explains, "nobody would ever be able to score, because the Kinect's estimation of your movements is not going to look the same."

The solution: filters.

The Harmonix Filter Tester Team is made up of dancers, both professional and amateur. They take the dance data from the animators and attempt to dance to the routine in front of Kinect, while recording their routine to plug back into the system. These recordings are then layered — or filtered — over the original animation by the people behind the curtain, known as filter authors. The filter authors "seed" the game with the first batch of data from the filter testers, and then the filter testers start over, dancing and recording, dancing and recording, until the filters have bridged enough of the gap to make being judged by Kinect as fun as it can possibly be.


"It becomes a game of ping-pong," says Sandel, "where we send it to the authors, they tweak it and send it back, we play it and check the score and send it back, they do some tweaks and send it back to us, until eventually we agree it's good enough. It's not too hard, it's not too easy. The players will enjoy playing this song. Then we ship it."

Stoddard compares the process of creating filters to the Japanese game show where contestants have to hold certain poses to fit into holes in a moving wall, or else be knocked on their rear. The Kinect can see certain shapes and certain poses. The process of creating filters is similar to creating the holes in the moving wall, so that when you play the game, if you dance the same way the dancer is dancing on screen, you won't get knocked on your butt by the game's inability to tell what you're doing.

"The idea of getting immediate feedback as you're playing the game as to how well you're doing," says Stoddard, "nine times out of ten, that correlates to how much fun you're having. If you know that you're doing a bad job and you can see why you're not being scored in the way that you'd like, and then you can correct as a result of it, the loop of that feedback is a very pleasing thing."

the score

Here's what we've learned so far: A hit song blows up the tubes of the internet and Chris Rigopulos reels it in for Dance Central. Naoko Takamoto then cracks the whip on a team of artists and technicians to convert that song into a dance number for the game. Devin Woolridge straps on his spandex suit and Adam Repka and a team of animators capture his choreography with LED lights, cameras and computer software. Then Andy Bouchard steps in to animate that captured dance and make it look nice, at which point Stoddard and Sandel and their team of QA filter tester dancers play the resulting routine over and over and over until it's fun.

That, in a nutshell, is the process of turning "Gangnam Style" into a Dance Central routine, as we are led to understand it by around the seventh or eighth hour of our time at Harmonix. The rest, we are told, is magic behind a curtain that we will not get to see.

And then we're escorted behind the curtain. Sort of.

Rachel Dixon works about 10 feet from animator Andy Bouchard, not behind a curtain at all. As it turns out, she's been hiding in plain sight all day, making magic right in front of us, and we almost didn't notice. Statuesque, energetic and light-spirited, you'd never know she was the keeper of so many secrets.

Rachel Dixon works on the Harmonix Authoring Team. Her job is to take the animated dance from Bouchard's department, seed it with data from the filter tester team, and then ping-pong back and forth with Stoddard and Sandel until the dance is ready to ship. Dixon, in short, is the person who decides where you will score points and how you will be judged by your Kinect as you try to recreate the dance in your living room. She is, in other words, the score keeper.

"I get a lot of data of what people look like dancing our dances and I ensure that the scoring is fun and accurate," Dixon says. "That you get a fun response when you get that 'Flawless' rating. Or if you need to work on your moves. ... Getting all that data makes sure that 'Awesome' feels awesome."

Dixon majored in dance, and worked for Harmonix for years as an administrative assistant until the company decided to make a dance game. That's when she got her chance. She was tapped to leap the fence and join the development team and entrusted with making magic. It is the definition of a Cinderella story.

"They pulled me up and said, 'You are probably detail-oriented as far as dance goes," she says. "'Help us zhuzj this system.' So I did."

I ask her what that word means, "zhuzj."

"'Mold,' perhaps? I think it's from Queer Eye for the Straight Guy."

Dixon laughs and you can't help but laugh with her. If she says zhuzj is the word, then zhuzj is the word. If she's having fun, you want to have fun with her. It makes perfect sense that she's the person who decides when a Dance Central dance is fun enough to play.

"I can tell if they're doing it right," she says. "I tell the machine if they're not flawless, if it's not the best that I've seen. It does a lot of statistical crunching, figuring out what the range would be, and gives you the magic of that flawless experience.

"There is a particular response that you get from trying to get something, almost getting it, and then getting a 'Flawless' or getting an 'Awesome' for the first time. That's inherently fun."


Dixon will dance the routines herself. Near her standing desk is a dance pad and a Kinect. The entire Harmonix office is full of these dance stations, in fact. Easy to overlook amidst the general sprawl of equipment and mess, but they are there. Like the state-of-the-art motion-capture studio hidden in the basement and the various other pieces of NASA-grade tech left lying around.

Finally, after an entire day at Harmonix, it becomes clear just what this company's secret sauce might be: At Harmonix, the equipment isn't the star of the show. It's just gear. For all of their investment in technological wizardry, it is first and foremost a company that understands the value of people. No piece of equipment will ever be more valuable than the admin who just happens to be a dancer, or the singer looking for a job or any of the other ridiculously intelligent and talented people who, with a whole wide world of opportunity for brainiac technology wizards and scary smart creative talent, have chosen to make video games about music.

"It's a fun play on words. It's the Wednesday dance move ... because it's Hump Day, yo! So that's my greatest achievement in all of Dance Central."

No one person exemplifies this freedom of spirit more than Rachel Dixon. In spite of having the final say on whether or not a dance is good enough to ship, what Dixon is most proud of is the fact she named a dance move.

"It has a variation in 'Gangnam Style,'" she says. "It's the 'Wednesday Dance' move."

I ask her to demonstrate and she does. It's basically the move where you thrust your pelvis forward and back, forward and back.

"It's a fun play on words. It's the Wednesday dance move ... because it's Hump Day, yo! So that's my greatest achievement in all of Dance Central."

Dixon collapses in giggles and the entire room fills with laughter.Babykayak