Inside the research that could change the perception of kids, gaming and gender

But the games industry will remember Wiseman for the presentation she made at this year’s Game Developers Conference in San Francisco. In it she shared the results of a survey she and a partner, Ashly Burch, conducted of more than 1,000 middle and high school-aged kids which seemed to overturn many long held assumptions in the games industry.

Polygon sat down with Wiseman and her team to find out more about her findings, and where her research goes from here.

Settle down, class

Wiseman has been a teacher for nearly 20 years. That gives her a unique perspective on adolescence and the social troubles associated with it, both in school and out. Her work is based on her experiences in the classroom, but also on her day-to-day efforts in schools around the country where she regularly talks with large groups of students in 6th through 12th grades about conflict resolution.

Imagine your entire middle school or high school being divided by grade level, or perhaps even grade level and gender, and then cycled through an hours-long presentation in the auditorium. There on the empty stage is an eloquent woman who could be your friend’s mom. And she’s there, you’re told, to talk to you about bullying.

What in the world are you doing here? None of your friends know, but at least you all got out of algebra class for a while.

The agenda for Wiseman’s presentations is simple: Arm kids with better ways of resolving conflict, more civilized strategies than cliquish exclusion, name-calling or physical violence. But getting to the part of the show where someone, anyone in the audience is paying attention to her is the biggest hurdle every time.

Rosalind Wiseman with a student focus group, from RosalindWiseman.com.

As Wiseman terms it, there is a crisis of credibility for adults that work with kids. If the kids don’t feel that you understand them and their culture, they tune out. She might as well be showing them slides from her last vacation. Worse yet, there’s the possibility that a bad delivery can discredit the ideas she’s presenting and actually do more harm than good.

So the strategy that she’s taken to, since about 2011, is to use video games as an icebreaker.

"One of the biggest challenges that we have in our work is that kids think any kind of conflict resolution program is stupid and cheesy and never going to work," Wiseman told Polygon. "I’m always looking for ways to integrate gaming imagery into the presentations I do, especially with middle school kids. I’ve been doing this for a while. I would put these images up just to get an emotional response from the kids, so their brains would trigger, and then I would talk to them intellectually, pre-frontal cortex kinds of stuff.

"I would show imagery of games and kids would start screaming with joy. I wasn’t even having any video, because I couldn’t show video because I’d get in trouble with the schools. But just putting up a picture of Master Chief would get kids high fiving, jumping up and down — totally engaged.

"Those moments of being with 400 kids screaming in joy because they see a video game character — you can’t put a price on it. It’s literally invaluable."

"400 kids screaming in joy because they see a video game character — you can’t put a price on it."

That got her foot in the door, so to speak. But still, there were moments where she felt a bit like a fraud, where her audience probed for cracks in her facade.

"I’m sitting there with an auditorium full of boys and a kid raises his hand — I’ll never forget this," Wiseman said. "He raises his hand like his hand’s going to pop off his body, and he says, ‘Are you PlayStation or Xbox?’"

Wiseman very nearly panicked.

"I had zero clue what he was talking about. Zero clue. I mean, I sort of did, but I really didn’t. Somehow, magically, I don’t know how, I said the only thing I could say in that situation to get me out of that, which was, ‘I’m not going to respond. Because if I do then you’re going to judge me.’

"I don’t know how I knew how to say that. He responded, ‘That’s fair.’ That was a big moment."

Wiseman realized that video games were a powerful part of these student’s lives, and came to learn how communities within schools formed around different games. The cool kids played Call of Duty, while the nerds played Minecraft. But at every presentation, she ran the risk of appearing out of touch or as a kind of carpetbagger from the adult world clad merely in the trappings of the Master Chief.

So she called in some help, in the form of well known games personality Ashly Burch, part of the family team behind the series Hey Ash, Whatcha Playin’?

Let's play

Burch, who has grown up in video game culture from an early age, became a sort of consultant for Wiseman, who was planning a companion book to Queen Bees and Wannabes, focused on adolescent boys, called Masterminds and Wingmen. With Burch's help, the book was published in 2013 and even included an entire chapter on gaming.

"How could I write a book about boys and not do something on gaming?" Wiseman said. "I didn’t know very much about it, so I started looking for somebody who might be able to help me. That’s how I found Ashly.

"She has been indispensable."

"She just needed explanations," Burch told Polygon. "What’s a MOBA like? I hear this game League of Legends. What’s that game? What’s it all about? How does it play? That kind of thing."

Ashly Burch

With Burch’s help, and the help of Wiseman’s assistant Charlie Kuhn, the team was even able to extend the gaming metaphor throughout her entire conflict resolution presentation. It was a strategy that resonated especially well with young men.

"We talk about a better way to handle yourself in conflict," Kuhn told Polygon. "Bullying happens. Conflict is inevitable. That sort of stuff. When that happens, here’s ways you can think about it in a different way.

"What we started to do was integrate video game imagery and how you would prepare for a boss level or to get through a difficult mission. In a game, what do you do to get ready? And we would tie that to conflict in real life."

It was surprising, to both Wiseman and Kuhn, when those same kinds of analogies were resonating with the young women that they talked with as well.

When Wiseman presented at schools she was always on the lookout for student editors, teens in the trenches — so to speak — to fact check the language surrounding themes and realities she wanted to write about in her books. Queen Bees and Wannabes needs to be updated every five years or so to keep pace with the changing social norms of the age range, so alongside larger presentations Wiseman says she gathers small groups of students and just has a conversation about topics she’s interested in.

One day, after one of those smaller focus groups broke up, something unexpected happened.

"No one in the world thinks that this girl plays GTA5. Nobody."

"This girl came up to me who was a cheerleader. A blonde girl, just a generically pretty kind of girl," Wiseman recalled. "She turned to me and said, … ‘Well, you know, I’m not like a 'gamer,' but I love Call of Duty and I love Grand Theft Auto 5.’"

Wiseman, somewhat taken aback, pressed her for more information.

"I just was looking at her and I’m thinking to myself, no one in the world thinks that this girl plays GTA5. Nobody. This is just not the stereotype that you think of. You make these kinds of cultural judgments. If you looked at this girl you would think that she’s the girl who sits next to her boyfriend while he plays Call of Duty."

This was the first bit of anecdotal evidence Wiseman had that there was a deeply rooted misunderstanding among adults of how young men and women engage with video games.

Around that same time, Wiseman said she found herself on the fringes of the brushfire that came to be known as GamerGate.

Information warfare

Starting last year, Wiseman said, everywhere she looked on social media and in the gaming press she saw evidence of women in the games industry being attacked. At the same time, it felt like GamerGate was putting words in the mouths of the students she came in contact with on a daily basis, imposing attitudes and desires on their gaming habits from the outside.

The bottom line, Wiseman said, was that GamerGate was promoting a kind of narrative around gender issues in gaming that didn’t match up with what she was seeing in her daily interactions with kids. But she was torn over whether or not to throw herself into the ongoing culture war.

"I really struggled, because I felt like, to me, there is this thing that I was experiencing in schools around the country that was a huge disconnect between what I was seeing in the gaming world," Wiseman said.

"Now I was reading the websites. I was paying attention to [GamerGate]. There was this enormous disconnect between the boys I knew who were playing, the girls I knew were playing, and what I was reading about. That was really a big deal for me.

"I really struggled with, do I have the right to speak about this? How can I contribute?"

She decided that, with Burch and Kuhn’s help, she would create a survey for students in middle school and high school, and try to measure their attitudes and feelings about what kinds of games they liked to play. Together, they built the survey up over several months, and then sat down with a research professor — someone with experience running these kinds of studies professionally — to polish it.

"We wanted to find out what games girls are playing," Burch said. "What are they interested in? Do they share the same concerns that we have about how women are represented in games? I wanted to know where kids are coming from.

"We were in this bubble with GamerGate that the kids weren’t even aware of. Outside this bubble of harassment and genuine horror that we experience through social media, what are girls actually interested in? What do they want to see in their games? What games do they enjoy playing?"

The survey was distributed widely, to professional contacts of Wiseman’s as well as teachers in schools around the country. Before long, entire classrooms and even whole grade levels were responding. The online survey reached sixth graders in Massachusetts and Alabama, and clear across the country it was taken by juniors in California and Arizona.

Before it was over, the team had received more than 1,400 responses from more than 11 different states.

The results, shown here and available online at the GDC Vault, were extraordinary. Among other things, the survey seemed to show that girls were much more invested in the gender of their character than boys were, and that while girls and boys seemed to play similar games at similar rates the term "gamer" didn’t seem to resonate with young women as much as it did with young men.

But what does this survey really tell us, statistically? And how much should the industry rely on it?

The ivory tower

Justin Patchin is the professor Wiseman took her survey to before she shared it with students. As co-director of the Cyberbullying Research Center, and a faculty member in the criminal justice program at University of Wisconsin-Eau Claire, Patchin makes his living studying youth through similar surveys.

"Myself and my research partner have been studying cyberbullying and social networking problems for years," Patchin told Polygon. "We know Rosalind from her earlier work, trying to guide teens in making responsible decisions and interacting respectfully with their peers. I’ve presented with her at various conferences.

"She was interested in trying to quantify some of the ideas she had about gaming and other things. My role in helping her was basically to be a sounding board in her efforts."

Patchin said that the group that Wiseman surveyed was a "convenience sample," or basically whoever she could get to take the survey. Yes, they came from a mix of public and private schools and a wide spread of different economic backgrounds. But they also just sort of showed up. Who’s to say if they’re a representative sample?

University of Wisconsin Eau Claire, from uwec.edu.

"The last eight-plus surveys we’ve done," Patchin told Polygon, "have been random samples of known populations in schools. This is where I tried to talk with Rosalind, where I tried to convince her about how important generalizability is."

"One could ask, ‘Is that school or district representative of the population of the United States?’" Patchin said. "The way we address it in our research is by trying to administer our surveys in a variety of schools all across the United States. It’s not perfect, but it’s better than going into one school and surveying 50 kids from that one school."

It’s not necessarily about the quantity of responses, Patchin said. It’s about the quality of those responses.

"Sample size really only matters with respect to the number of variables you’re using in a model to explain things. [Wiseman's team is] not using multivariate analytical models to explain these things. They’re just looking to get percentages. … They’re basically looking at raw numbers and percentages."

Nevertheless, Patchin says that this kind of research has to start somewhere. His own work on cyberbullying began with the same kind of convenience samples that Wiseman has collected information from.

"From my understanding, she found some pretty interesting findings," Patchin said. "So now, of course, the next step is to replicate that and do another test in another school. Maybe 100 researchers can take questions and administer them in other populations and see if they hold up. That’s the scientific process."

"This kind of research has to start somewhere."

Discussions are ongoing to have some of Wiseman’s more interesting questions tacked onto one of Patchin’s upcoming, more rigorous surveys. Then, with an academically applied statistical analysis, perhaps we can get closer to finding out the truth about adolescents and gaming. But the results of his work are quite a ways out.

While Wiseman and her team created and administered the survey over the course of just a few months, Patchin’s survey will take a long time just to get it in front of students.

"I’d love to say it will be very soon, but the reality is … we’ve been trying to administer in this particular school district since October" Patchin said. "For the results to be published? That could be at least a year afterwards."

Level up

Wiseman's team admits that probing student populations for this kind of information is challenging, mainly because of the layers of bureaucracy in American schools.

"We understand that this is not going to stand up to the rigor of an academic survey," her assistant Kuhn said. "We’re not going to pay administrators to deliver this survey. We’re just curious about this stuff. Let’s see who’s out there, who’s willing to take this."

"In a private school setting you have more control over what you can do with your students," Kuhn said. "Sometimes this was teachers in their classrooms ... Or it could be a parent on Facebook. It could be someone on Twitter finding this link and us saying, hey, this is great. Some daughter or cousin or aunt saying, please take this on behalf of this person. Then we did it more systematically. That’s when we ran into a bit of the bureaucracy and difficulty of getting into public schools. That was a bit frustrating.

"The research we were trying to do was only to help inform schools to do things better. But due to systems and process ... We weren’t allowed in. And so I think that the number, as lots of people might say, is a decent sample size. It should be bigger. Shoulda coulda woulda. I’m proud of our 1,400 respondents, because it’s something that’s very difficult to do. We scrounged those together. We put some time into it, but it also came to us."

The trouble is that gaming, just like other forms of technology, is changing faster than our ability to statistically record its perception among today’s youth.

"This is the biggest problem with the work I do," Patchin said. "Because if technology changes all the time, cyberbullying changes. The same would apply to games. From the time I collect the data to analyze results to the time something actually appears in a peer-reviewed journal, it’s probably at least a year. Which is extremely frustrating."

That means Wiseman’s study, even as a hastily done convenience sample, is still important. These kinds of questions weren't being asked of this population before, and now they are. And researchers like Patchin have reason to take them further.

To Burch, the findings are more than just the first steps on the way to better research. They validate her experience, and that of women like her.

"I’m friends with so many girls and women who’ve played games their whole lives," Burch said. "I had a feeling this was probably going to be interesting information to present, because it was going to show a different picture than the one we’re used to seeing. It felt like it was more important than ever to get that information out there.

"It felt like it was more important than ever to get that information out there."

"The thing I’m happy about with the GDC presentation, and the survey results, is the size of our sample. Would have been amazing if we could have gotten 10,000 kids? Absolutely. But with 1,400 kids, coming to the conclusions that we came to, I thought was so encouraging.

"But it was also so encouraging to see that boys aren’t the way that we tend to characterize them, and that girls aren’t the way we tend to characterize them. Everything is so much more complex than how people tend to assume. Either for good or for ill, people tend to dilute how kids are. Kids are a lot more complicated than we give them credit for. There’s a lot more to kids and gaming than we consider. Getting us to have this conversation. It was important, especially with the climate of GamerGate, to give girls a voice, because there was so much silencing happening."

Note: For reference, we've included Wiseman and Burch's complete slide deck from GDC 2015. You can also view their presentation here.

Wiseman and Burch GDC 2015 study