How Gearbox's 'Truth Team' outwitted Borderlands feedback

Gearbox's Matthew Armstrong and Stephanie Puri spoke at SXSW Interactive about the strange contradiction that the focus testing process represents. Puri works on Gearbox's "Truth Team," a unit dedicated not only to the management and collection of focus testing feedback, but also the oftentimes sideways interpretation of that feedback.

Focus testing is critical in just about every part of a video game's development — but sometimes, the feedback garnered from those tests is flat-out wrong.

Borderlands director Matthew Armstrong and Gearbox's Stephanie Puri spoke at length about that contradiction during their panel at SXSW Interactive. Puri works on Gearbox's "Truth Team," a unit dedicated not only to the management and collection of focus testing feedback, but also the oftentimes sideways interpretation of that feedback.

"A couple of things were very clear to us when we started working on Borderlands, and a few things took a few tests for us to really understand," Armstrong explained. "There are rules that we've come to interpret as key things we need to follow, things like: Testers try to speak in fact, but they speak in emotion."

That sounds like semantics, but it's a very important distinction. For any number of reasons, be it personal bias, predisposition towards a particular franchise or whether or not the test subject is hung over, the tester's feedback could be applicable to themselves, rather than the game being tested.

"They speak about their experience, not what actually exists in the game," Puri added.

The second rule: Testers do not like features they don't understand.

"We get feedback that says this is awful, this is terrible," Armstrong explained. "The truth is that they may not be terrible, it's just that the tester didn't fully grasp what it is they're looking at or experiencing."

Finally: Testers have expectations that may not match the developer's.

"They speak about their experience, not what actually exists in the game."

"For instance, Borderlands is a game about wanting things," Armstrong explained. "But one of the common things we hear people say is 'Boy, I'd like to build my own gun.' Okay, you can build your own gun. Now the game's over, congratulations. The quest for the perfect gun is over. It ends when you can build your own gun, and if you can do that in the first hour of the game, the game's over."

These three fallacies led the Truth Team to come up with a few tricky fixes for common complaints from focus testers. For example, early testers reported that the sprint speed was too slow. Rather than increase that speed, Gearbox designers simply scattered more debris on the ground, and increased the player's field of view. Players passed by objects with increased frequency, giving the illusion of speed without requiring any huge adjustments to the play mechanics.

A similar complaint was that guns took too long to load. The designers knew this wasn't a real concern, as reload times were a balancing conceit for weapons, and even the slowest weapons reloaded quicker than some in Call of Duty (the "benchmark for a shooter that feels good," says Armstrong). Rather than speed up the reload, Gearbox added more motions to the reload animation, again giving the appearance of heightened speed.

The real wisdom of Solomon, however, came from user's complaints that Skag Gully, one of the game's earliest areas, had too many eponymous Skags. Players reported that they were running into too many clusters of enemies while moving through the Gully, offering feedback like, "this isn't fun, this is boring."

Gearbox, as a result, tripled the number of Skags in the area.

"All of a sudden, it wasn't a travel area that had too many enemies getting in the way," Armstrong said. "It was a combat area."

Armstrong and Puri were careful to explain that the Truth Team doesn't just ignore focus tester feedback. Rather, it analyzes the intention behind that feedback for the most effective solution.

"In that example, the problem wasn't that there were too many Skags, it's that the pacing was bad," Armstrong said. "But the tester might not have known how to say 'The pacing is bad,' so we had to figure out what they really meant.

"If you interpret everything your testers are giving you as straight fact," Armstrong added, "our solution would have been to remove all the Skags, and we would have had a game that was slightly more boring."