How facial expressions could be the next game controller

I don’t usually care for infinite running games, but In the Same Boat is a bold exception. It’s an experimental project in which players use facial expressions to control a canoe, as it navigates obstacles along a river.

I recently played the game at the University of California, Santa Cruz, where it’s being partly developed by researchers and students as a joint project with the University of Saskatchewan. The team is part of a program run by Katherine Isbister, author of How Games Move Us, an excellent book on emotive game design.

Here’s how In the Same Boat works: A camera is placed on top of a PC, which monitors my facial expressions. On-screen emojis prompt me to laugh or frown or grimace. If I time my expression correctly, the boat progresses.

But that’s only half the game. In the Same Boat is a two-player game, so I’m connected to a another player who is doing the same thing as me. We have to coordinate our facial expressions in order to beat the game.

At first, I find it odd to simulate facial expressions, but after a few mishaps and triumphs, I find myself having a lot of fun. In fact, I begin to find it difficult to frown at the appropriate times, because I’m genuinely laughing.

The game begins to feel less like a goofy experiment, and more like a genuinely smart piece of work. It makes me feel the way I do when I play a fun new board game with my family.

The game’s goal, according to the game’s academic documentation, is to create “intimacy, trust, fondness, and affection over a distance through networked play.” Its use of coordinated action leads to “a heightened sense of social closeness between participants.”

“To the best of our knowledge, this game is the first to leverage physiological syncing over a distance,” states the game’s documentation. “It has the potential to help geographically-distributed friends, family, and partners to feel closer through playful interaction.”

In the Same Boat’s underlying tech is partly based on the work of Raquel Robinson, a former student at UCSC. Her project, All The Feels, is a biometric and camera-based Twitch overlay that analyzes how viewers are emotionally connected to the streams that they watch. This kind of real-time feedback isn’t merely a way to capture data, but a part of the streaming experience. It’s easy to imagine streamers responding to the biometrics that they inspire.

Isbister’s work on social and emotional technology ranges far and wide. During my visit I watched a presentation about a laser tag-style LARP game which turns the genre on its head by rewarding players for acts of kindness and generosity, within the framework of a narrative in which there’s a genuine “win” state. It points toward a future for games that isn’t so wedded to conflict as today’s commercial model.

“Lately, tech has taken a lot of hits for driving us apart and polarizing us,” said Isbister. “I believe games that bring people closer and that build emotional and social connections are not only extremely fun, but also could help bridge the gaps and mistrust that separate us.”