Meet the people trying to make games accessible to the blind and partially sighted, and the gamers who stand to benefit.
Brandon Cole still has vivid memories of his first gaming experience. His brother handed him a Nintendo Entertainment System controller and told him to press start. "Before I knew it, I was smashing bricks, collecting coins and beating every single level," he recalls.
But there's a catch. Cole wasn't playing Super Mario Bros., like he thought. His brother was. Cole held the second controller and pressed buttons while his brother completed the entire game in single-player mode. The joke was even more cruel than it sounds, because Brandon Cole is blind.
But for Cole, it planted a seed. "Some part of me realized how much I loved the experience of 'playing' a game like that," he says, "and some part of me knew it was going to happen again. Before I knew it, I had vowed to keep trying video games until I beat one all by myself." A few years later, after countless attempts, he played the entirety of Killer Instinct for the Super Nintendo unaided.
Excited and triumphant, Cole never looked back. He had crossed a threshold into the sighted world and mastered a game. Other console triumphs followed, with titles such as Mortal Kombat and Big Hurt Baseball ("the best audio of any [Super Nintendo] game I've personally ever heard") proving playable to the unsighted Cole.
His passion lives on to this day. "I continue to try new things, I continue to surprise myself and my appreciation for games and the industry continues to grow," Cole explains. Yet the game industry at large does little to welcome Cole or any of his fellow blind gamers. PlayStation 3 and Xbox 360 provide mazy menus and ill-distinguished interface sounds that confuse even the fully sighted — and that's before someone even gets into the games themselves.
There are some trying to bring change to how blind people experience games, however — to open the slew of yearly big-budget releases and breakout indie hits to the millions of people who can't see their flashy graphics but love to hear and touch and play all the same.
Guitar Hero and Rock Band took the world by storm with their plastic instrument rock star fantasies, but their reliance on visual cues proved troublesome for blind gamers hoping to get in on the color-coded rhythmic fun.
Enter Rock Vibe, a blind-accessible music game by Rupa Dhillon. It translates Rock Band's scrolling fretboard-style on-screen prompts into vibrations — often referred to as haptic feedback — on a special belt, with the latest version taking a MIDI file as input instead of using the original game.
Rock Vibe puts blind and sighted players on equal footing, Dhillon says, "because it's not something that you normally do with any game or in life." Despite the name and origins, it's a radical departure from the experience of Rock Band and Guitar Hero.
"When it's visual, things can come at you much faster," Dhillon says. "You know that it's coming, and at the same time you can add different visual cues ... holding a note, for example. You can't really do that with tactile feedback because you overload the player. It's not something that they're used to. And when you overload the player with too much stimuli, they get frustrated because it's really, really difficult."
So it's not perfect. But it's fun, and challenging, and it opens music games to a broader audience of blind and partially sighted players than the handful determined (and skilled) enough to figure out the note patterns by ear.
But Rock Vibe may not reach a particularly big audience. Dhillon raised $17,400 on Kickstarter to produce it for a select group of donors, testers and organizations that work with the blind. She makes it all herself — a one-woman production and development team — so costs are high and availability is limited.
Eelke Folmer knows from his own experience that accessibility is more than just a matter of building interfaces anyone can use. He and a student in his department at the University of Nevada collaborated on Blind Hero, a blind-accessible version of Guitar Hero, in 2008. Unlike Dhillon, they used a glove to provide haptic feedback. It cost almost $2,500 to make each one.
This high price is not unusual in technology for the blind. "I go to these conferences on assisted tech," says Folmer, "and they show these gloves that can do things like assisted Braille. But these gloves are like $20,000; it's really great stuff, but no one will be able to use it because it's too expensive.
"A lot of assistive technology like screen readers — they're very expensive. Screen readers are like $1,200, so there's a consideration there as well. We need to make it accessible."
Folmer turned to cheap and affordable parts bought off the shelf. He worked on VI Tennis, a Wii Tennis clone for the visually impaired, then added VI Bowling and an original Whac-a-Mole game called Pet-n-Punch to this VI Fit ensemble. These games require only a Wii remote and a PC with Bluetooth.
This gesture opened the games up to a broad audience of visually impaired people around the world. Folmer reports approximately 10,000 downloads of VI Tennis since its release in 2009, peaking around Christmas each year when parents buy Wii remotes for their blind children.
"There's a lot of people in my field who [were] totally anti-games, so that's kinda hard. And now that's really changed."
Folmer and his students have since explored ways to make Microsoft's Kinect more accessible as well. "We developed a technique where you basically use a screen grabber and you look at certain visual cues in the game," he says. "Not all of the games, but Kinect Sports, for example — when you do the hurdles game, you run and one of the hurdles starts to light up green and then you jump."
A PC hooked up to this screen grabber watches for the changing color, then sends a vibration to a Wii remote held by a blind player. "So they'll be running the hurdles game, and then they'll feel haptic buzz and they'll jump."
Now Folmer's branching out and experimenting with new ways to utilize these technologies. "We've done another game that's basically — it's a little bit difficult to explain, but we have developed a technique for blind people where they can acquire spatial information — they can acquire a point around them, with their own hands.
"We have developed a game for that and did a test this summer [at Camp Abilities, a sports camp for blind children] to figure out how well they can grab, like, virtual apples around them."
Most recently, Folmer's been working to apply his research in game accessibility to other areas. A project called GIST puts a Kinect sensor on your chest, where it tracks your hands and provides spatial information about your environment — the distance to an object your finger is pointing at, or its color, or whether you're pointing at a human, for instance.
Microsoft provided a $25,000 grant to develop GIST further. Folmer can't wait to try Google Glass, too. "I'm just really excited about companies like Google and Microsoft who come up with this cool technology that has the potential to transform the lives of people with disabilities," he says.
He's noticed a trend. "There's a lot of people in my field who [were] totally anti-games," Folmer explains. "So that's kinda hard. And now that's really changed because in human–computer interaction, games have become a driving force of development. Things like the Kinect and stuff like that. The newest technology is all coming from the domain of games."
Games are driving technological innovation in accessibility, but they're also changing things at a grass-roots level. Brandon Cole and many others blog about their experiences as blind gamers, gradually spreading awareness about their plight. Some take a different approach.
Liam Erven makes Let's Play videos of console games on YouTube and develops mobile and computer games for the blind in his spare time. Like Cole and an estimated 39 million others worldwide, he's blind. "Some people think that I'm doing this for attention," he explains, "and as I tell people, I'm like, 'Look, if I wanted attention, I don't think I'd fake being blind, because being blind sucks. There's nothing fun about it. It's just not a good time.'
"I've talked to some people off the channel and once they realized that I was real they got really interested," he continues. "People's biggest question is, 'How do you do this if you can't see?' And it's such a hard question to answer because it's like I just taught myself to do it."
Erven was born blind, but he played games from an early age. "My family is one of those families that, even if you are blind, they still treat you equal. I was still expected to do everything," he explains. "So the line between being blind and sighted kind of blurred and so as kind of a lark they bought me a Nintendo."
This was 1989; he was around 4 years old, and he played Mario all day. "I couldn't get very far," Erven admits, "But to me the concept was interesting and I found it fascinating that you could do this. So I, for whatever reason, always got every new system as it came out."
Like Cole, Erven soon found games he could play. "My favorite was Track and Field 2 by Konami," he says. "It was really cool because a lot of it was based on timing." Years later, he discovered the same fact about Punch-Out!!. "I found it really interesting that you didn't have to look at the screen so much because everything was based on patterns.
"So for Glass Joe, you worried more about, 'OK, let me think about the uppercuts, let me know when that's coming' — I have a sound tell. Or, 'I know he's going to throw a certain number of punches,' versus games today [which] actually have an AI where the AI kind of anticipates what you're going to do before you even do it."
Erven pursued his hobby with vigor. "I was the first at Blockbuster in '92 when the first Mortal Kombat came out for Sega," he recalls. "I didn't really know what it was, but I just had the feeling that it was something I would enjoy. And I did. I loved it."
He would be there again 19 years later for the midnight launch of the series reboot. He's also a huge fan of PaRappa the Rapper, as well as a skilled Rock Band player — thanks no doubt to his learning cello, clarinet and accordion in school. Hearing him talk about his love of video games, Erven sounds just like any other self-professed gamer.
But his inability to see excludes him from many games. Final Fantasy X was the first in Square Enix's famed series to be voiced, and still it requires huge amounts of memorization or sighted assistance for the blind to play, while Pokémon has yet to receive a single entry that's even remotely blind accessible.
Even text and menu-sparse games, like those in the Mario and Sonic franchises, present a seemingly insurmountable challenge for a blind person (although Erven's first attempt to play Sonic 4 is impressive at times).
This is where Let's Play videos come in. Erven can share his unusual perspective on popular games, educating his audience on how they work in the mind of a blind man, and he can get a taste of games as the sighted experience them.
"There's a guy that I really really like, and he's actually a really cool guy outside of the arena of Let's Play-ing," says Erven. "He goes by newfiebangaa; his name is Clint. And he reads all the text on screen, which is fantastic." Not only will he read it, Erven adds, "but he'll change the voice for each character."
"It's a mindset of people. People choose not to understand. They choose to ignore what they can't understand."
Let's Plays are a window into the world of sighted gaming for guys like Erven, who understand the lingo but can't see the screen. "Me and my friend have a Dropbox folder that literally is 96 gigs of mp3s of different Let's Plays," he continues.
"We just get into it because it's so cool that these guys will read stuff — with some games like Mario there's not much to read, but it's cool to hear them describe stuff," explains Erven. "Some guys are really, really good at that.
"And you can enjoy it more because instead of just having the game audio with no one talking, you actually have an idea of what's going on. They may not describe every little thing like, 'Hey, I just hit a red block and something came out,' but you know more than you would if it was just straight audio."
Erven enjoys making his own Let's Play videos, playing favorites like Space Channel 5 and Tekken 3 from start to finish. "The thing that I explain to people is, 'Look, blind people are like every other person; they just can't see,'" he says. "We have a multitude of interests."
Erven rifles off a list of blind friends who are into cars, anime, gardening, skydiving and even rafting. "There's kind of like this — I want to say belief — that because blind people are blind they don't have the same [interests] that people with sight do, and I don't think that's true," he adds. "I think that's the biggest thing that has to be dispelled.
"It's a mindset of people. People choose not to understand. They choose to ignore what they can't understand, so instead of maybe taking the time to really think about how this would work for somebody, they just brush it off and go, 'Whatever,' which is unfortunate."
Erven takes care to put his best face forward, because he knows he's representing blind people everywhere. "What I tell people is if you're blind what you do directly affects what people think about everyone else in that community," he says.
He travels an hour and a half every day on the train to and from work. He talks to a lot of people, doing his bit to show how blind people are just people who can't see. This includes explaining that he plays games. "Why not?" he says. "Gaming has sound. You won't see me playing Pong. But gaming is something that we can do."
Finding a voice
Brandon Cole's website is dedicated to showing that blind people can — and do — enjoy mainstream console video games. "I have not only helped existing blind gamers with the site, but I've also created new ones," he says. "I've gotten several emails talking about how this person or that person got a console because of me and knew what games to try, and better yet, how to play them.
"I've even gotten emails from blind gamers I didn't know who had seen my website and were playing things even I didn't realize were playable." Cole blogs about the latest games and Microsoft, Nintendo and Sony news from his perspective as a blind gamer, honing in especially on sound effects and voice acting — his favorite topic.
"At first, voice acting in games was just something neat," he explains. "It wasn't particularly good when it started, but I still held games that had it over games that didn't simply because, 'There's talking!'"
Cole cites Interplay's 1995 PC role-playing game Stonekeep as an early standout, but it wasn't until the PlayStation 2 era that his obsession bloomed. "A couple [of] things were responsible for that," he says. Metal Gear Solid 2, the Grand Theft Auto trilogy, Enter the Matrix and Kingdom Hearts all contributed to a merging of Hollywood and games. "Here was a bunch of Hollywood talent in a video game," he continues, "and even those characters who weren't Hollywood talent were voiced by really good voice actors."
That's the root of it all for Cole. He loves movies and acting like he loves games. "I've actually wanted to be an actor ever since I was very young," he explains, "and [I] still do to this day, though nowadays I'd be willing to settle for voice actor rather than film and TV."
Cole has developed an uncanny ability to recognize not only people but also the method of acting — voice and motion together or separate — just from hearing a short clip. "The performances we're getting now are beyond anything I could've imagined even 10 years ago, when I had already started noticing that voice acting was a thing in games."
Now he's thinking about how he can break into the scene. Cole has experience, "though nothing professional yet." He's been in audio dramas and audio games, and took lead roles in three junior high plays back in his school days. "So I have the beginnings of a background," he says.
"What I don't have is super awesome mic equipment, or a good recording environment or a way to teleport to and from LA so I can audition for the many, many games that have their voice work recorded there. Yet still, the dream lives on, and [it] will continue to do so even if I never get the chance to achieve it."
I hear fragged people
AudioQuake — listen below to hear it.
You wouldn't think of Quake as being the most blind-friendly game around, but id Software's fast-paced first-person shooter was one of the first mainstream titles to receive a blind-accessible makeover. Matthew Atkinson and Sabahattin "Sebby" Gucukoglu were first-year university students in 2002 when the partially sighted Atkinson introduced Quake to his blind friend.
"I still remember first trying it for myself, with [Atkinson's] instruction, and using the axe to discover the presence of walls," says Gucukoglu. "When the axe was swung at a wall, it would make a clank." Grenades bounced and rolled off walls, gunshots had a satisfying echo and a variety of grunts and groans emanated from players' speakers.
The pair decided to take Quake the extra mile from "very nearly accessible" to completely accessible to the blind, working gradually over several years to alter the source code (which id released in 1999) and add the necessary features.
"We modified the gamecode of Quake [the part relating to how it behaves] to include the same sorts of 'mobility aids' as a blind person might have, or ideally have, in the real world," explains Atkinson. "This included detection of when they were directly facing an obstacle [and] to warn of changes in the environment such as slopes. Also we had a sort of radar-like effect for enemies that were visible on-screen and for items such as health packs, buttons that could be pressed and ammo."
And so was born AudioQuake, an adaptation of Quake specifically geared toward blind and partially-sighted players. Both its creators fondly remember the early tests between them, pitting sighted against blind in a one-on-one deathmatch. "The first ever accessible game of Quake played over a network involved Sebby running towards me swinging an axe," Atkinson reminisces. "I will not forget how that felt!"
Apart from a community-made Star Wars-themed mod called Jedi Quake, which encourages use of manual weapons over automatic guns, AudioQuake went largely untouched from 2006 until recently. A serious road accident put Atkinson into intensive therapy, and the game was pushed aside. Now nearing completion of "an unbelievably lucky recovery" that sees him edging closer to restoring his independence, both he and Gucukoglu recently dove back into the game — eager to work with the still-dedicated community to bring it up to speed on modern machines.
"My best estimate of the number of blind players active in the Alter Aeon community at any given time is around 300."
The blind gaming community has also amazed other developers who worked to make their games more accessible. Alter Aeon is a multi-user dungeon — otherwise known as MUD — started in 1995. "My first memory of a blind player came a couple of years later, when it was mentioned that ASCII art caused problems, and that certain things were hard to handle because of the spam level," says lead developer Dennis Towne.
"I found the concept of playing with a screen reader to be really amazing, and while I never expected it to be common, I did think it was worth trying to support." His team started creating settings for blind players, then later put together a separate blind mode.
It paid dividends. "My best estimate of the number of blind players active in the Alter Aeon community at any given time is around 300," says Towne, "with 40-60 blind players on at any given time. On average, [at] least 75 percent of our player base is blind, and the percentage has been climbing as we grow."
Ian Humphreys, better known online as Spoonbill Software, found similar results when he rewrote his freeware cribbage game after a client complained that it didn't work with screen readers. Humphreys first created hundreds of audio files of him speaking each sound snippet that might be needed, then discovered Microsoft SAPI — a speech synthesis and recognition system — could be used instead of manual recordings.
"Things blossomed from there," he says. "At the moment I have about 3,500 blind and vision-impaired clients on my books, but I know for a fact that this must just be the tip of the iceberg, because many players just pass the games on to their friends without going through me."
Better-known developers have also gone out of their way to improve their blind support. Niels Bauer added blind compatibility options to turn-based space trading sim Smugglers 3, then developed Smugglers 4 and 5 with blind players in mind.
"I realized that it would be a very worthwhile project to create a blind compatibility mode," he told GameSpot in 2009. "Not financially, but it was just the right thing to do." It's no small task to include text labels for every button in a deep, complicated strategy game, but the blind community latched onto it — with hundreds of messages about the Smugglers games on blind gaming mailing list Audyssey.
A common talking point among blind gamers is the field of audio games. These are, as you might imagine, video games without graphics. They first emerged in the mid-'90s, with Doom-clone Shades of Doom and racing title Drive being perhaps the most broadly covered in the press, and have recently found a welcome home on Apple's iOS devices — where polished efforts like Papa Sangre and BlindSide are making a mark on both sighted and blind players.
"Back when we were doing Drive and the early projects that followed, audio games were simply a novelty for, well, pretty much everyone, us included," says AudioGames.net and Creative Heroes co-founder Richard van Tol. "Games for the blind in the form of text adventures and other ASCII/text-based games had been around for quite some time, but games based on sound, well, there was only a handful of them."
By 2008, when Jason Allen started work on his first game, an audio-only roguelike called Entombed, audio games were only slightly less of a novelty for the sighted, and still in relatively short supply for the blind. Allen more or less stumbled into the field. "My wife's father has a condition called retinitis pigmentosa," he explains. It's an inherited disease that leads to significant or total loss in vision, usually later in life.
"He was complaining about not being able to play solitaire anymore. They got him a new monitor and it was higher resolution, so the cards were a lot smaller. So I thought, 'Wouldn't it be cool if I just completely made this game audio?'"
Allen found AudioGames.net in his search for help getting started. "[But] I never actually made the card game," he confesses. "Instead I went straight to making this role-playing game called Entombed."
Two years later, he finished the project — thanks in large part to assistance from the community, which ended up not only providing feedback but also went out and made sounds for Allen to use. Allen freely admits the game's many flaws, but he speaks with admiration for the audio gaming community that embraced it nonetheless.
"I got an email one time from a father telling me that [for] his daughter, who's just a teenager, [it] was the first time she had been able to use a computer," Allen says. "She was just playing the game. She could completely play the whole game, and she'd never really used the computer by herself, he told me. And that's just brilliant. I love that."
Allen recently left a job in software development to make indie games full time. He's focused on mobile for now, but he'd like to switch back to audio games soon. The audio games community "pretty much love[s] anything you give them," he says. They're constructive in their criticism.
"But then you go and create a mobile game, where 99 percent of the community hates you and maybe 1 percent will say, 'Well done; good job.' You get hate mail. You get bad comments. It's an unbelievable difference." Making games for the blind is "just better for your soul."
When Liam Erven was a kid, he created a superhero alter ego called Super Liam. "I remember being as a kid, sitting down and writing out, OK, working out this guy's world, who his enemies are, what he did," Erven says. "I think the proudest moment of my life so far has been taking all those ideas and putting them into an audio game that people could download and play.
"In general, the response has been incredibly positive. The one thing we found was that the visually impaired tend to get through it much more quickly than sighted players."
"It was beautiful. And people will look at it and go, 'Super Liam' and they'll scoff, but to me it's like one of my crowning achievements that I could do something like this and take an idea and OK, I couldn't do it in a full video game but I could do it in an audio-based game. At the time that I created it I was the first one to do a side-scroller. So I was the first one to do a platforming game for the blind using stereo audio."
Erven has made several games at this point, including Super Egg Hunt — which involves running around on a grid collecting beeping Easter eggs, trying not to get caught by an angry chicken — and most recently Audio Archery for iOS. He's focused on simple experiences, and he's innovating on what works in an audio-only setting.
This is key to the growth of audio games as a medium, according to van Tol and his Creative Heroes colleague Sander Huiberts. "Many audio games base their gameplay on existing video games," van Tol explained via email. "From a blind-accessible game perspective, this is only logical: everyone wants to save the universe. However, not all types of gameplay lend themselves very well to the audio-only medium."
Aaron Rasmussen and Michael T. Astolfi didn't quite avoid the "save the world" trope with their award-winning audio-only horror game BlindSide, but they made a big impression by having the protagonist wake up blind and need to feel (and bump) his way out of his apartment and around the world outside. (The idea came from an experiment in Rasmussen's high school chemistry class. Rasmussen wasn't wearing goggles, and went blind for days after a red phosphorous and potassium chlorate mixture literally blew up in his face.)
"In general, the response has been incredibly positive," Rasmussen said in an email interview with Polygon. "The one thing we found was that the visually impaired tend to get through it much more quickly than sighted players."
Van Tol and Huiberts have seen this phenomenon firsthand. "When sighted gamers started to play [Drive], they initially had a hard time to focus on the game," says van Tol. "Even though they knew they were playing an audio game and nothing would appear on the screen, they were simply waiting for something to happen and were not listening at all."
Out of sight, out of mind
Audio games such as BlindSide and blind-accessible video games like those made by 7-128 and Spoonbill Software are very much the exception to the rule, though. Visually impaired and blind players are often left behind by an industry obsessed with making money and pushing out technological advancements.
What would it take to bring them into the fold? Not much, really. Both Erven and Cole stated voice-over support — especially in the main console interface — as their main wish. "I would love my Xbox to give me audible feedback as to what screen I'm on, what menu I'm in and what the items are," says Erven.
"I don't think it would take a lot of investment. I think the problem is that it's just not a feature that's really wanted," he continues. Apple pushes accessibility right out of the box on both its iOS and Mac devices. But Erven thinks that's a charitable move — it's not making any serious impact on the company's bottom line. "Sometimes I think companies just have to be altruistic and kinda step up and go, 'We want to make people's lives better.'"
Cole has another wish, in addition to built-in screen readers. "What I envision is a universal button sequence or button command," he says. "Something like, 'Hold both triggers and press up down up down,' which then activates a game's accessibility mode. This would then be all we would have to memorize, and the sighted players need not even be bothered by seeing the accessibility option in their menus."
For developers, the lesson is that audio can be a part of the core mechanics of a game, and it's worth investing in right at the beginning, in the conceptualization phase. Sony and Microsoft have new consoles coming out later this year, which means it's a chance for a clean slate with blind accessibility. But nobody's getting their hopes up. "Back when the first Kinect came out, a bunch of us asked Microsoft a bunch of questions about it," says Cole. "As expected, the people we asked didn't really have answers for us.
"So what it's going to come down to here is probably what it usually comes down to, which is that someone, probably me since I preordered both consoles, is going to have to try both thoroughly, figure out what's what and if it is more accessible, then write up something to give everyone else an idea of what's what."
Brian Bors from the Accessibility Foundation is less optimistic. Awareness is building among designers, he notes, but consoles are moving away from the turn-based and slower-paced experiences conducive to voice-over support. "I think that a total cure for blindness is actually a more realistic proposition than a total blind-accessible mainstream game," he says.
For now, the blind community waits with bated breath, hoping that maybe this time it'll be invited to the party. "I'm really looking forward to some of those games," Cole admits. "So right now if I can't use either console, I just don't want to know."
Images: Michelle Ciotta, Rupa Dhillon, Phil Lamperski, Matthew Atkinson, Sabahattin Gucukoglu
Editing: Matt Leone, Russ Pitts
Design / Layout: Warren Schultheis, Matthew Sullivan