Ubisoft Montreal is feeding its writers criticism sandwiches

To describe Jonathan Dankoff's job in the most digestible way, it's hard to do better than his Twitter bio, which explains in a sentence that he tries "to help people make their games better."

Officially, Dankoff is a user research project manager at Ubisoft Montreal. And for the last five years, he's been focusing on Assassin's Creed at the studio behind games like Assassin's Creed 3, Assassin's Creed 4: Black Flag and Assassin's Creed Unity. About a year ago, he started specializing in something new. Along with his partner, Corey May, he now examines story through user testing. And all that really means is that he takes the criticism and suggestions that a bunch of people have and shares them with Ubisoft's writers before it's too late.

Ultimately, Dankoff, who talked about what he's learned at a GDC 2015 session this week, sees himself as a helper. And as he was working on games in development, he realized something not that long ago: Narrative needs help. For reasons even the new process' creator doesn't understand, narrative was never part of the rigorous playtesting process that has become a hallmark of video game development. He decided to fix that.

Game and mission design receive constant feedback, but story gets "next to none," he said. And that doesn't make any sense, given his understanding that narrative has become the second most important component for game reviews.

Narrative tends to becomes fully integrated into games late in the development process, he said. At that point, scripts are locked, motion capture is done, voice over is recorded. Identifying story problems late into a project can make them somewhere between difficult and impossible to fix. So, to Dankoff, the solution seemed to be identifying these problems before it's too late.

As everything does with hindsight, his process seems obvious. But it's new at Ubisoft, a company whose flagship Assassin's Creed series is steeped in lore.

Now that he had a plan, a logical question followed: How do you formally review narrative? They couldn't use standard user testers because they "are not critics," Dankoff says. Their criticism often boils down to "Oh, I don't really like this part." And that's not particularly helpful for figuring out how to change a game's story and make it better.

So he and May devised a solution, based on the belief that there's value in iteration. Playtesting shows that. The process of building a game and testing it with insiders and outsiders to find and fix mistakes is an integral part of making games. Narrative, they decided, needed the polish that gameplay got.

That's why they devised the narrative review process, where project writers become Ubisoft's clients.

Dankoff and May insert themselves into the process, calling attention to narrative in the messy middle where it's sometimes ignored. Their job is to listen to mixed signals and filter them for writers, who can use that feedback to make better games.

At Ubisoft Montreal, they assemble a team made of two distinct parts: peers from the writing community as well as those who are "specifically not writers," like game design, level design and other leads.

Identifying story problems late into a project can make them somewhere between difficult and impossible to fix

The formal structure of the process is deliberate, Dankoff said, because everybody tends to have ideas about how to change and fix stories, and they all think it's an easy fix. When he said that narrative creators tend to get "crazy feedback from everyone on the development team," the room of assembled writers broke into knowing laughter.

Through a systematic process, they review the narrative, analyze it and synthesize what they learn into a report. Their job is to "read, write, analyze and discuss," he said.

Participants have 10 days (and, likely, an extension beyond that, because everybody's always so busy) to read a story synopsis, take notes, ask questions and provide notes. A longer questionnaire finishes the process. At Ubisoft, it's a very basic web form. You don't need "a fancy digital questionnaire," he told the audience.

Next up is the analysis phase, where they look for consensus as well as divergent opinions, like comprehension of the story's continuity and character motivation. And then it's time to provide that information to the writers, in the least messy way possible.

Dankoff warned against the multiplicative effects of negativity and suggested anyone employing this process should and end with positivity, saving the criticism for the middle. Think of it as a criticism sandwich.

A formalized discussion follows. The writer can now ask questions of the reading group. And they can employ brainstorming sessions to workshop the problems people have identified. The results go into the first report as an addendum.

The point is to provide meaningful feedback for a very important part of a game that hasn't been given the proper amount of feedback. They want to be early to act and discover problems and then provide a clear direction for fixing problems. They want, ultimately, to help everyone invest in the narrative, when it's easy to make changes. They want to do it long before they'd have to get a voice actor to come back in the studio.

Their narrative review process provides actionable feedback, and the document that comes from the process becomes ammunition for writers, when people want to change the story.

Before he ended the talk, Dankoff made one thing very clear: The narrative review process is not story design by committee. "What I'm talking about is accountability," he said. It's a process that allows writers to "justify your actions much more easily."

The process is "not data-driven," he said. It's "data informed." Reviewers, in other words, don't write a script for anyone.

None of the games Dankoff and his writer cohort May have worked on for the last year have come out, so he was careful not to give away any Ubisoft secrets. But he believes the process already provided benefits. There is a writer (on an unspecified game) who had no idea that a problem existed until the review process pointed it out. Under the old process, that kind of thing might've been swept under the rug and uncovered when it was too late to fix. Another example took the opposite approach: A writer knew there was a problem, and the group worked to improve the part of the script without overriding the writer's intentions.

He said he couldn't believe they were the first people to think of this. That's why he came to GDC: to share what now seems simple and obvious with fellow developers.