Description of graphical content is included between Description Start and Description End. Transcript Start [Silence] Fade up from black. Animation: Text for TSBVI transform into braille cells for TSBVI. Fade to black. Fade up from black. [ Slide start: ] Description Start: Title: Developing Spatial Navigation Skills through Video Game Play Content: center photo: elementary age girl plays a video game using a keyboard. Reference: Lofti B. Merabet; The Laboratory for Visual Neuroplasticity, Massachusetts Eye and Ear Infirmary, Harvard Medical School Description End: Merabet: My second presentation today a little bit different. I would talk to you now about some projects that we've been doing previously before we got very, very heavy into the C-V-I aspect. This was a large scale study about five, six years, [ Slide end: ] looking at video game use in blind children-- blind individuals in general-- ocular blind-- to try to develop navigation skills and orientation mobility skills. So very, very different in terms of what we talked about earlier this morning, but nonetheless trying to come towards this using this evidence‑based neuroscience, neuroscience-driven approach. And hopefully, you'll have questions for this as well. And I am available to stay and discuss also about the C-V-I talk, as well. So let's, let's get started. In the same way that I started off my first presentation I kind of want to get sort of the lay of the land with you, and try to give you a sense of how, how the thought process came about for this project. And understand that I'm going to show you in about four or five slides what I was thinking about for like three, four years so-- [Laugh] To try to compress all that. [ Slide start: ] Description Start: Title: Rehabilitation: Way Finding Content: left-side text: Orientation & Mobility (O&M) Training left-side photo: view from waist down; man and women walking with white canes, man also has a guide dog. Description End: So here, here's the first thing to think about, rehabilitation in the case of way finding and navigation. Obviously a big, big challenge for all the people we work with, right? [ Slide end: ] So fortunately we have a structured way to teach people with visual impairments how to find their way around. We call that, of course, "orientation and mobility instruction." Everything from a cane to a guide dog, for example. All very structured, well established techniques are really part and parcel to promote an individual's independence. There are limitations, of course, with this and there are always people and O&M instructors reaching out to me saying, you know, "What do you think about this technology? What do you think about this approach?" And so on. "Is there a way that we can study this and incorporate in a, in a more structured fashion?" And I became very, very interested in this idea. [ Slide start: ] Description Start: Content from previous slide and center photo: Dan Kish riding a bicycle in a parking lot Description End: Some other individuals, Dan Kish for example. You've probably have heard about this guy. He uses echolocation. They call him the, "the Human Batman." He walks around making click noises and using the, the reflections off of, off of the surfaces of objects, he's able to identify various objects and in this particular case you see him, riding his bicycle, even though he has prosth-- he's, he has prosthetic eyes. He has absolutely no light perception. [ Slide end: ] I don't know if everybody can learn this skill. It's certainly really quite remarkable and there have been some groups in Canada who have done f-M-R-I in him and studied his brain and how he's able to do it. But it's, it's quite a, quite a remarkable skill that, that he's developed. [ Slide start: ] Description Start: Title: Content from previous slide and right-side photo: college age woman waking with a white cane in her right hand while reading a refreshable braille device with her left hand. right-side graphic: drawing of the earth with a ballerina on top, label, "GPS." Description End: Some technology that I think is quite interesting as well. This is an interesting one. This is from the Sendero Group out of California. And the idea is that you walk around with the G-P-S monitor, which tracks you, and as you're walking through the city you connect this with, say, your braille notes, you get information about, for example, the name of the street, how far you are from a particular destination, you may use a bluetooth connector as well to get some auditory input. Some, some very, very nice technology that's coming together to, to help enhance these skills, if you will. Certainly limitations. [ Slide end: ] The big one with G-P-S of course is that it's only for outdoors, right? G-P-S doesn't work in an indoor environment. It also is quite limited when had you're in a situation of being downtown, where there's a lot of reflections from buildings and so on, satellite doesn't capture. You have to be visible in order for this, for this to work. So we were thinking about what was out there, what could we change? In particular we were very, very motivated or trying get to this idea of motivation, I should say. How could we leverage motivation as a way to improve navigation skills? So let's, let's, let's talk about a, a few things as well. First point I want to make from a clinical rehabilitation standpoint, a general comment that, that I'd like to make with you. [ Slide start: ] Description Start: Title: Traditional Clinical Rehabilitation Therapy Content: left-side text: In a traditional therapy session, the patient works one on one with a therapist to address specific goals (e.g. psychological issues, movent foci, skill) left-side photo: man seated on a sofa talking to therapist with back to the camera. upper right-side photo: older woman seated in wheelchair reaches for object held over her head by a therapist. lower right-side photo: woman is seated at a table with a child's stacking toy. center text: What is the ecological validity and the effect of context on theraphy? Description End: So in traditional therapy session the patient works one-on-one with a therapist to address specific goals like psychological issues. Could be movement foci, or a particular skill, in the hopes of improving that particular deficit or that particular function. So, for example, if a person has a phobia or particular psychological issue they work one-on-one with a therapist, voicing those concerns, walking through those issues and trying to make that one-on-one face time, that exchange to, to work through that, that issue. If you are working on the motor side, very, very often what we see is a lot of repetition, right, working with various tasks. A particular skill of motion deficit that's trying to get enhanced through repetition and repetitive exercises and so on, right? [ Slide end: ] That's-- You all agree with me that's sort of like the state of the affairs right now. Here's my problem, all right? A couple of things to, to think about. What is the ecological validity and the effect of context on therapy? If I'm sitting with a therapist talking about my problems and I'm not having the problem how good am I at transmitting that issue? Right? Similarly if that therapist is providing me some strategies and I'm still not going through that problem, how good am I in terms of transferring that into that situation, into that, in that scenario. So the context, the immersion of learning the skill is extremely important. That's the first thing I want to say. [ Slide start: ] Repeat previous slide The second aspect if we look on the motor side of things, right, boredom kills us when it comes to rehabilitation. Right? Everybody recognize this, this toy? Right? This little stacker thing? I don't know about you, I was three when I had one. Right? [ Laughter ] Here's a woman who just had a stroke in her forties struggling to do something that she knows was designed for a three‑year‑old. [ Slide end: ] What does that do in terms of her motivation and struggle and so on? So I really think the ecological validity and the context of therapy is extremely important. We certainly can do better. So the immersion aspect, I think, is extremely important and creating scenarios that are meaningful for that individual are also extremely important. All right? So let's get to some other pieces of the puzzle and I'm slowly going to edge into this idea of gaming and how we got into that. [ Slide start: ] Description Start: Content: left-side graphic: info graphic titled: "Exactly How Much are the Time A-Changin'?;" various statistics for traditional and digital telelcommunications. right-side graphic: Bizzaro cartoon of a elementary age children walking and texting; yellow street sign with warning, "Slow, children texting." right-side text: Next generation Description End: The times are changing, definitely. Right? For example here. Daily emails: 2,000, 12 billion emails being sent. We're now 247 in 2010. Text messages: 400,000 up to 4.5 billion. Right? Time spent online: 2.7 hours a week to 18 hours a week. Moral of the story: we are a tech-driven society. Right? [ Slide end: ] A lot of what we do is intimately related what we do with technology? And as I said, despite my, my early screensaver problems, I believe that technology is an enabler and we should try to leverage that somehow. And we're getting very, very good at it because costs are going down. Everybody has a cell phone, everybody has email, everybody has ways to stay connected. There's an opportunity here that I think we, we need to, to leverage. [ Slide start: ] Description Start: Title: The Case for Play Content: photo montage; children dancing, dressed in astronaut costume, toy room with drawing pad and playhouse; two panda bears playing; cover of The New York Times Magazine, titled, "Why Do We Play?" Description End: Couple of other things to talk about, the case for play. As I mention my mother is a preschool teacher and she used to always tell me "a child who plays is a healthy child." Right? It's intimately related. And indeed play is extremely important in the development of a child. [ Slide end: ] Role playing, social interactions, what's fair, what's not. Establishing rapport with kids. All that is done at a very, very early age and I think that's also what makes games later in life very, very exciting as well. It's a way to be somebody in a sense that you can't otherwise be. Animals know this. Right? Young animals play fight. And they know when they can bite, when they can't and so on. So there's something very important about play and brain development that I think is very, very interesting. [ Slide start: ] Description Start: Title: The Case for Play Content from previous slide and right-side text: High/Scope Educational Research Foundation - Longitudinal Study (Stuart Brown) - By age 23, more than a third of kids who had attended instruction-oriented preschools had been arrested for a felony as compared to fewer than one tenth of kids who had been in play-oriented preschools. Description End: And think of the counter-example, this is a child with autism. It's a child who doesn't play. Right? And that's one of the hallmark signs of a child with autism as well. So I think there's somehow an association between playing and brain development and so on. And many, many news stories that have been out there trying to get at this point. Here's an interesting study. I don't know if you've heard about this one called the High Scope Study. This was done in the state of Michigan, done by the Educational Research Foundation in Michigan. A longitudinal study by Stuart Brown. So what he found that by age 23 he compared individuals who went through a very, very structured, didactic school program versus schools who had very, very a lot of hours of playtime and interaction. And what he found by age 23, more than a third of the kids who had attended an instruction-oriented preschool, had been arrested for a felony as compared to fewer than one-tenth of the kids who had been in a play-oriented preschool. [ Slide end: ] Now. Doesn't mean you don't play, you're going to rob a bank, right? [ Laughter ] This isn't causality. But a very, very interesting association that having this early on in development certainly seems to have a benefit for brain development as well. So those questions earlier on about C-V-I, how do I wake up the visual brain, consider play as, as one of the ways to do it, from an engagement standpoint. And I hope to convince you that there's a neuroplastic and a neuroscience reason behind this as well. [ Slide start: ] Description Start: Title: The Case for Learning by Simulation Content: left-side photo: inside the cockpit of an airline simulator. right-side photo: woman in lab coat using a laparoscopic surgery simulator Reference: Transfer Effectiveness Ratio, 48 (Johnston, 1995) Description End: Learning through simulation, another piece of the puzzle, very, very important. The best example to give you is flight simulators. If you are a pilot, wanting to learn how to fly a new plane or how to land at a new airport, or how to fly in very, very challenging conditions, much better that you do this in a simulator than a compliment of 350 people behind you. Right? If you make a mistake, better you learn it there, than you do it in the real world. Right? [ Slide end: ] So pilots spend an enormous amount of time in flight simulators and this has been extremely effective and has revolutionized the airline industry. And they have something what's called the transference effect ratio. Which is about 50%, which means every two hours that you spend in a flight simulator is the equivalent of one hour of real flight time. So what you learn in the simulator, going through the motions, preparing yourself mentally, serves into the real world. And the closer that immersion is, the better it is in terms of the transference. Also people have learned this, surgical, for example, the medical field is spending a lot of money looking at surgical simulations. Better I make a mistake resecting a tumor in a sur-- in a simulation than I do in the real world. The military is also spending a lot of money on this as well. So learning by simulation seems to be another thing that the brain likes. And again, I will show you some evidence of that. [ Slide start: ] Description Start: Title: Video Games and Virtual Reality in Therapy Content: left-side photos: elementary student with wearing virtual reality googles; screen shot of animated characters crossing a street from the video game, "Street Safety with Buddy and Friends," screen shot of animated characters walking on the sidewalk. Reference: Elizabeth Strickland; http://www.digitalspace.com/projects right-side photo: screen shot of virtual reality soccer goal with video insert of student in a wheelchair as the goalie. right-side text: Interactive Virtual Reality Exercise System (REX) right-side photo: product photo of REX system with monitor and cpu. Description End: Here's some great examples of how video games and virtual reality are being used in therapy in, in your own world. This is work by Elizabeth Strickland. What she has is she has children with cognitive-- children with cognitive development issues. And trying to teach them basic skills like crossing the street, safely. So she has these kids wearing a, a virtual reality helmet and they do associations, right? They call this game called Street Safety, they associate good behaviors with certain friends, bad behaviors with other individuals and they go through these simulations learning to cross safely. [ Slide end: ] Better you learn this in the safe, controlled environment of a classroom than learning this in the real world-- the hard way, so to speak. You learn these skills in a safe, controlled environment. You have reinforcement, you have repetition, again things that the brain likes. And then you transfer that to the real world. So very, very interesting approach. [ Slide start: ] Repeat previous slide Another one that's quite nice this is, called I-REX, this is gesture talk. This is a group out of Israel that have developed an interesting system. This is a child with cerebral palsy who doesn't want to go to rehab. Right? Sorry, sounded like Amy Winehouse there. [ Laughter ] How do you get him to go to rehab? No, no, no. [ Laughter ] He goes, "Well, what I do like- what I do like is soccer." Right? So he has the system here where they use a small camera, they film him, they project it on a blue screen, and he's the goalie while people take shots, and the idea is that he reaches over one side, blocks the ball, reaches to the other side, blocks the ball. Then it can go systematically, crossing the hemi field, the other-- the other hemi space as well. [ Slide end: ] And all of this is quantified as you can see on the bottom row there. They've got his favorite team playing, his favorite players are playing, he's engaged, now he wants to go and he's engaged. So again, you can do good work with play and under simulation, this is a chance to try to awaken the brain and motivate individuals. [ Slide start: ] Description Start: Title: The Case for Video Games: A Parallel Track of Education? Content: left-side photo: Two adults playing first video game, Pong; graphic advertisement for Pac-Man; two boys playing race car video game; package cover for World of WarCraft. right-side text: World of War Craft - since 1994: collectively gamers have spent 5.93 million years playing - average gamer spends 22 hours/week ...a part time job? Description End: Let's talk specifically more about video games, why I think-- whether or not I think this is a good idea. Probably you all remember when Pong came out. [ Laughter ] We thought this-- Oh, my God, I gotta get Pong, right? [ Laughter ] This is revolutionary, right? Now think about how games have evolved, right? [ Slide end: ] It's really interesting. We've moved outside of the arcade and now moved it to our own personal devices, so it's really interesting that the goals remain the same, but the space that we work in has changed dramatically. [ Slide start: ] Repeat previous slide. Here's some interesting statistics. World of Warcraft, which is a, a role-playing game. It's a very, very interesting one because they actually log on the time that people spend. And here's some interesting stats. Since 1994, collectively gamers have spent close to 6 million years playing this game. [ Slide end: ] That's geological time scale, right? [ Laughter ] The Grand Canyon was built in 6 million years, right? [ Laughter ] One game. Right? And the average gamer spends 22 hours a week. [ Slide start: ] Description Start: Title: The Case for Video Games: A Parallel Track of Education? Content from previous slide and center text: Countries with strong gaming culture - by age 21: average gamer will have spent 10,000 hours = time from 5th grade to high school graduation, if you have perfect attendance. Jane McGonigal. right-side photo: Book cover, "Reality is Broken" by Jane McGonigal. Description End: That's a part‑time job, right? These people spend a lot of time playing video games. Another interesting book called Reality is Broken by Jane McGonigal, she's a, a, a game designer and also a, a sociologist-- very, very interested in that. And she says in countries with strong gaming culture by the age of 21, the average ga-- the average gamer will have spent close to 10,000 hours playing video games, which is the equivalent of time you spent from the fifth grade to high school graduation, if you have perfect attendance. [ Slide end: ] That's a lot of time spending in front of the monitor. So they're doing it is what I'm trying to say. Can we leverage this somehow? Right? [ Slide start: ] Repeat previous slide. Other things: are video games useful from from a rehabilitation standpoint. I'm going to give you a couple of other recent examples. Wii-habilitation, you're all familiar with the Wii-mote? Right? This idea of a, of a-- it basically has a gyroscope in it and it can sense directions in three axes of, of motion, and translates that onto, onto a monitor as you interact. Some interesting work with stroke recovery, again, [ Slide end: ] I don't think the evidence is very, very clear how positive it is. Could be a huge placebo effect of just simply being engaged with a group and so on. Which may account for some of the benefits that are there, but indeed people are studying this and looking how to engage individuals. The social interaction, for example, as well. [ Slide start: ] Description Start: Title: Are Video Games Useful? Content from previous slide and right-side graphic: newsclip of headline, "The Impact of Video Games on Training Surgeons in the 21st Century." right-side text: "a link between skill at video gaming and skill at laparoscopic surgery. current video game players made 31% fewer errors, were 24% faster an scored 26% better overall than their non-player colleagues." Description End: Wii Bowling, for example. And a lot of situations. Promoting you know, social interaction and so on. A lot easier to do this in a virtual living room than it is to actually take them all to a particular site. So there's some benefit in that as well that I think is, is quite interesting. Another interesting study here: The impact of video games on training surgeons in the 21st Century. There was a link, and I quote here, "a link between skill at video game-- video gaming and skill at laproscopic surgery. Current video game players made 31% fewer errors, were 24% faster, and scored 26 better-- percent better overall than non‑player colleagues. [ Slide end: ] Again, not causal. Doesn't mean that you should be playing video games and then go to medical school. [ Laughter ] The point is that there was an association between the two, right? So, an observational study. So something again kind of connecting from a skill standpoint. And the last one I'll share-- I was really, really struck with. [ Slide start: ] Description Start: Title: Are Video Games Useful? Content from previous slide and right-side graphic: newsclip of headline, "Predicting protein structures with a multiplayer online game." right-side graphic: screen shot from online game Foldit. right-side text: "using Foldit, the 3D conformational structure of a protein was solved in roughly 1 week by individuals with no specialty training in biochemistry." (http://fold.it/portal/info/science) Description End: This was a study that was published in Nature a couple of years ago. And to give you sort of a background, I, I'm not a biochemist. But apparently when it comes to figuring out the three dimensional shape of a protein or a molecule, it's really, really difficult, it's really complicated. It's like a mental teaser or puzzle and so on. Even with some the fastest computers it takes months and months and years to figure out this three dimensional shape. So this group of investigators decided to come up with a game called Fold It. And the idea was to go online, there were various rules of how you could fold this particular shape and they just kind of left it out into, to the world to see what would happen. And they said that using Fold It, the 3-D confirmational structure of a protein was solved in roughly one week. Right? By individuals with no specific training in biochemistry. [ Slide end: ] The best neur-- the best scientists were trying to figure out and couldn't do it with the fastest computers. And all of these guys went online, no interest in biochemistry whatsoever and they solved it in a week. [ Laughter ] So gaming somehow brings the best out of us, right? We think in a way that we don't typically think in more sort of didactic fashions. And I think, again, that's another aspect that I want to submit to you. [ Slide start: ] Description Start: Title: Are Video Games Useful? Content: center graphic: newsclip of headline, "Video-Game Play Induces Plasticity in the Visual System of Adults with Amblyopia (PLOS Biology) left-side graphic: Flow chart of video game process. right-side graphic: table with 4 columns and 4 rows of graphs demonstrating changes in acuity for test subjects. Description End: Another interesting example in our, in our field: This was a work by Dennis Levi, at the University of Berk- University of California Berkeley, using video games to try to improve Amblyopia, visual acuity. It was a very preliminary study. But what they found that a lot of the kids going through this and interacting with video games showed improvement in their visual acuity a couple of lines. [ Slide end: ] Again, largely an observational study, there were a lot of randomization and control issues, and I think it needs to be replicated. But showing you that we can take this also directly from the visual acuity standpoint or visual performance standpoint as well. Now, why do I think games work? I'm going to give you what I call my neuroscience rationale. [ Slide start: ] Description Start: Title: Games...a Neuroscience Rationale Content: - Atatinable Rewards...jewels, points, munitions, portals, "Epic Win," etc. - Task Novelty and Graded Difficulty...Attainable goals. - High Attention Demands...Survival Pressure...death, time constraints. Description End: So all video games have three really important aspects. Doesn't matter where it's PacMan, or World of Warcraft or something, they all kind of have three basic features. The first one is that there's always attainable rewards: jewels, points, munitions, portals, the epic win-- the epic win feeling and so on for my World of Warcraft colleagues. There's also task-- task novelty and graded difficulties. All games start off really, really easy, then they get a little bit harder, they seem to be almost perfectly paced with you. And they're just at the point where you don't give up. Right? You never just say, "I don't want to play this anymore." [ Slide end: ] You just "Okay, one more try, one more try, one more try." And, and figuring out that grada-- the gradation is obviously a big, big key. And this idea of having attainable goals. And last was they always have high attention demands. Right? Survival pressure, death. Right? Time constraints, monsters, all this sort of thing keeps you engaged obviously into the game, right? Well, what does this mean? [ Slide start: ] Description Start: Title: Games...a Neuroscience Rationale Content from previous slide and left-side graphic: label, "Reward: Dopaminerigic System;" profile drawing of the brain with different systems labelled. center graphic: label, "Novelty and Goals: Seratonergic/Noradrenerigic System;" profile drawing of the brain with different system labelled. right-side graphic: label, "Attention: Cholinergic System;" profile drawing of the brain with different system labelled. Description End: Well, first of all, reward is intimately related to dopamine, right? Novelty and goals is intimately related to serotonin and/or adrenaline. And finally attention is intimately related with acetylcholine or the cholinergic system. The point here is that we are wired for this, right? We like this. Video-- good video game designers know- understand our brain chemistry and in a sense are tapping into this to get us engaged. [ Slide end: ] And my argument is there's an opportunity here that we don't typically have. And how do we jump start the brain? This- this may be one way to do it. [ Slide start: ] Description Start: Content: left-side graphic: Title Graphic for "Doom" and embedded animation from the game; described by speaker. right-side graphic: faceless, animated figure sits on the floor playing a video game, raises hands in victory and thought bubble shows colored game map. Description End: So here is the study that I want to share with you. You probably remember this video game Doom, right? Came out early '90s. Yeah? I- I wasted years of my life -- [ Laughter ] Playing this game. It is really, really addictive for lack- for lack of a better term. It is amazing. It was one of the first games of its time, what's called a 3-D first person shooter game where you walk through a virtual labyrinth. I'm just going to show you a video if you're- if you're not familiar with it. [Electronic music and sound effects] They've got the loud music going on- very, very high-paced. You're walking through this three-dimensional environment, you gotta kill the bad guys, have to find your way back through the corridors. Very, very high-paced. Very engaged. You get a sense right away of what's going on here. All right, I'll- I'll- I'll spare you the violence. [Laughs] [Electronic music and sound effects ends] The point-- the point here is that to play this game, to succeed in this game you have to build a mental map in your mind of the world you're walking through. Right? You have to get a sense that "Okay, I walked through this corridor, I've been in this room before or I came in from another perspective." [ Slide end: ] So as you play the game you develop a cognitive map in your mind. Right? All right. That being said, I have a colleague from the University of Chile, Chile who's a university-- I should say, was a computer scientist and develops video games for blind children. [ Slide start: ] Description Start: Content: left-side graphic: title graphic "Audio Doom" center photo: profile image of boy at a keyboard in front of a computer screen. right-side photo: portrait shot, labeled, "Jaime Sanchez and Coworkers C5: University of Chile, Santiago." left-side graphic: label, "target virtual world;" graphical representation of game route with icons for doors, jewels, and obstacles. Description End: And the game he developed was Audio Doom, which is exactly the same thing, except it's based purely on audio cues. And I'll explain to you more into how, how that works. All right? So as you see here just like kids -- sighted kids, of their age- their peers-- these blind kids play the game forever. They love it. [ Slide end: ] They're completely engaged, hours and hours playing the game and the other thing he noticed from an observational standpoint was the kids who played the game were doing better in school. They seemed to be better at math, they seemed to be better at spatial reasoning, they were much more engaged socially with their peers than the kids who didn't seem to like video games. Again not causal, but an interesting association nonetheless. So the thought was, "Was there an opportunity here?" And there was another interesting piece that really sparked my interest when I- when I first saw this. [ Slide start: ] Description Start: Content from previous slide and right-side photo: label, "final representation;" picture of lego representation of the game route with doors, jewels, and obstacles. center graphic: label, "target virtual world (modeled on actual building); video game floor plan. right-side photo: label, "transfer of spatial knowledge to physical building;" high school student with white cane entering a hallway. Description End: For example, here, if I give you a target environment like this, so this is the labyrinth, the child comes in here, this is a door, another door, a dead end, they go through another dead end, a series of monsters and so on, and they gotta find their way to a portal that takes them to the next level. Right? If you give the child Lego pieces and ask them to build the map that they walked through, they can build a perfect one-to-one representation of the virtual world they walked through. Right? They have the map in their mind even though they've never seen the map, all based on auditory cues. And in fact these are congenitally blind children so they've never seen the world, period, but nonetheless they can build a map in their mind. So it- they, they, they can generate this through non‑visual cues. So the question is this: Why not play the game in a world that actually exists and use that as a way to teach orientation, mobility, and navigation? And that's exactly what we did. So we invented a game using the same sort of strategy and this is the layout of an actual physical building at the Carroll Center for the Blind. We had the kids play the game and the goal here is kind of like PacMan, you gotta roll through the building, you gotta find these little jewels that you see in blue squares, and I'll show you a video how the game is played. And you also have to be careful-- these, these red guys these are the monsters, right? And if they catch you with the jewel, they hide the jewel somewhere else. So it forces you to kind of keep expan- to keep exploring the building and so on. And you have to catch as many jewels as you can and not get caught by the jewels. So we engaged them to do this. They then play the game, then we physically take them to the building and say, "Okay, now that you have this map in your mind can you find your way." The important thing to keep in mind is at no time do we tell them this is the goal of the study. We just simply say this is a game, this is how you play it and then we see what happens. All right? [ Slide end: ] [ Slide start: ] Description Start: Content: Photo: aerial shot of Carroll Center for the Blind showing eight buildings nestled among mature trees. One building outlined in yellow. Overlayed photos: architectural floor plans of building; video game floor plans of building; Carroll Center logo; over-the-shoulder photo of boy playing video game on a laptop; photo of same boy with white cane entering hallway in the building. Description End: So a little bit more details about this. As I said, this was done at the Carroll Center for the Blind in Newton, Massachusetts a little bit outside of Boston. We chose this building here, which is the St. Paul building. The reason why is because this is an administrative building, or at least it was at the time, and the kids had no prior knowledge of the layout of the building there. It's a two story building. Has about- about 20 rooms. Allows us to do sort of a real world scenario, two floors, look at for example, interactions between floors and so on. And as I said they don't have any prior experience with this building when they come to the campus. They play the game, as I mentioned, we never say, you know, "memorize the layout" or anything along those lines. We just say "play the game" and then we take them physically there and we have a series of outcomes to see how well they're able to learn the routes. [ Slide end: ] [ Slide start: ] Description Start: Title: Audio-based EnvironmentSimulator (AbES) Content: left-side text: "earcons" spatialized audio. left-side graphic: drawing of a boy sitting in front of game screen with speakers on his left and right; compass graphic overlayed on video game floorplan; cardinal coordinate labels, "north, south, east, west;" speakers labeled, "left" and "right;" keyboard graphic with game keys labeled in red. right-side graphic: close-up image of game door with audio icon; cartoon image of child; image of game door in sound bubble off his right ear. right-side graphic: embedded animation of AbES video game. Description End: All right. So more details about how the game works. We call this AbES for "audio-based environment simulator" or "AbES." We don't have as clever a name as Audio Doom, but this is- this is how it works. So you're all familiar with icons, right? The wastepaper basket, for example, on your computer is where you put documents you don't like. So we use earcons, exactly the same thing. So to give you an example is a knocking sound. So I think I can play this one here. [Knocking sound] All right? If you hear that sound you know that that's the presence of a door. If I hear that knocking sound in my right ear that means the door is on my right side. If I hear the knocking sound in my left ear I know the door is on my left side. If I hear it in front of me this- the door is in front of me. Keep in mind that also when I'm walking through the environment and I hear that knocking sound in my right ear, if I turn around 180-degrees and come back I now need to hear the knocking sound in my left ear. Right? So what the software is doing is keeping track of your egocentric heading and presenting the sounds in a spatialized manner so that you can build the spatial map in your mind as you interact with it. Okay? So we use cardinal coordinates: north, south, west, east. So they can always kind of work in that rigid, in that rigid cardinal coordinate system, as I said. Left ear, right ear, either with speakers or with headphones. And every step they take is measured or scaled to an actual physical step in that building. So here's a video of, of a child playing the game. And remember, they don't see anything on the screen, right? This is -- I'm just simply showing you this so we can, so we can track them what the actual movement is. Here they go. Electronic Voice: East. [Electronic sound effects] Merabet: Knock on the door, open it, walk in. Electronic Voice: [Indiscernible] [Electronic beeps] Merabet: That's the warning sound that there's a jewel. As they get closer and closer to the jewel the loudness of the sound happens Electronic Voice: East Merabet: Allows them to get organized. Electronic Voice: South Merabet: They get oriented to where that sound is. [Electronic sound effect] Merabet: They get the jewel. Electronic Voice: East. They gotta go outside now -- take it outside. The red box, as I mentioned, that's the monster roving around, trying to catch you. [Sound effects] Electronic Voice: Stairwell one. East, Merabet: An obstacle. Electronic Voice: North. [Footstep sound effects] [Doorknob sound effect] Electronic Voice: [Indiscernible] Merabet: Outside, yay, applause, points. Electronic Voice: West. South. Merabet: Come back. [Doorknob sound effects] Electronic Voice: First floor. Stairwell one. West. Merabet: They're in the stairwell. As they climb the stairs, the pitch increases. Second floor. Electronic Voice: You're halfway east. Merabet: And they get to the top there. Electronic Voice: Second floor. Merabet: And they keep exploring, and exploring, and exploring, and they play for a total of about an hour and 30 minutes, an hour and a half. All right? Completely engaged, as I said. They see nothing on the screen. That's for us to track them to see where they're heading. And believe me, it's tough to get this out of their hands, they are really, really engaged in playing this game. [ Slide end: ] [ Slide start: ] Description Start: Title: Audio-based Environmental Simulator (AbES) Content: left-side text: AbES: Directed Navigation Mode left-side graphic: screen shot of game floor plan with user interface dialogue for starting point and ending point. left-side graphic: screen shot of game floor plan with specific path outlined in yellow; label, "Structured Learning." center graphic: cartoon drawing of child right-side text: AbES: Game Mode right-side graphic: screen shot of game floor plan with obstacles, monsters, jewels, doors and stairwells labeled. right-side graphic: Drawing of a scroll with game rules, "find the hidden jewels, bring them outside, avoid the monsters;" inset picture of cartoon character giving a thumbs up; label, "Self Discovery Learning." Description End: Here's the study design. As I said, we did this as a randomized clinical trial. So we took all-- all comers into the study. I'll give you more details. And we randomized them into three actual groups. And before I show you the three groups let me show you the breakdown of the various aspects. You can play AbES, the video game, if you will, in directed navigation mode. Right? This means that I give you a start place and an end place and you learn the, the layout of the building. And what we did is we pair each child-- or each individual in the study with an orientation mobility instructor who sits next to them and teaches them step‑by‑step the layout of the building. All right? The same way- a virtual replication, if you will, of what they would do in an actual O&M instruction of that building, all right? So they work one‑on‑one with an orientation & mobility instructor, right? So that's called structural learning or the directed navigator arm of the study. The other arm is the one-- really the intervention of interest-- is the gaming arm, exactly as I said: you know there are monsters, there's [Indiscernible] and so on, we simply explain to the child or the individual this is how the game is played, this is the goal, you gotta find these jewels that are hidden throughout the building, you gotta avoid the monsters, if they catch you, they hide the jewels somewhere else. And the more jewels that you can find, the better it is in terms of your score. We never tell them you have to specifically learn the layout of the building, we just simply say, "This is how you play the game." [ Slide end: ] [ Slide start: ] Description Start: Title: General Study Design Content: center graphic: Flow chart of study design; blocks representing parts of the design; top-center block, "Enrolled"; splits into three columns, "Directed Navigation," "Game," "Control;" screen shot of video game floor plan in each column; center block, "AbES training (3 x 30 min sessions);" center block, "Assess Proficiency of virtual navigation in St. Paul Building;" center block, "Assess Proficiencty of real world navigation in actual St. Paul Building;" center block, "Assess Proficiency of Drop Off Tasks (Problem Solving) in actual St. Paul Building." Description End: So now, as I said, this was a three-arm randomized clinical trial. We had three arms in the study. Some were enrolled or randomized to the directed navigator arm, again working with an orientation mobility instructor. Some were playing the game arm. And some were in the control group. So this was a game but the building had nothing to do with the target that we're trying to get. So we wanted to see the potential benefit of actually playing the game itself, even though the overall target wasn't matching. They go through, they played-- as I said for for about an hour and a half, each one arm. We look at their proficiency of virtual navigating, so going from Target A to Target B or Target C to Target D, and so on. We look whether or not they can do it and how long they take. We also then transfer them to the real world, we take them to the physical building and see whether or not they can use those transferred skills, what they learn in terms of their map. And then the last thing that we look at is what are called called "dropoff tasks" which you are all familiar with in the O&M world. So in other words, instead of asking to just go A and B, C and D, E and F-- we bring them to various positions in the building and we say, "Where you're standing now, what's the shortest way out of the building?" Right? So we give them sort of a task to force them to manipulate the information in their mind. So that's the dropoff task in this. So just to remind you, the comparison of direct navigation versus game tells us something about the method of instruction. Right? The comparison of the control group versus the game tells us something about the gaming context, that's why we have the three arms. Okay? [ Slide end: ] [ Slide start: ] Description Start: Content: left-side text box: Inclusion Criteria - 18-45 y.o. male+female - documented legal blindness by age 3 -blindness of ocular cause (all subjects blindfolded) center text: text box: Outcome Measures Quantitative: - number of paths correct - time to target - "creativity points" Qualitative: - type of errors made - strategies employed right-side photo: blindfolded player and trainer at laptop computer; label, "Training with Facilitator." left-side text box: Stoping Rules: - max time of 6 min per trial - 3 successful, successive trial completions to continue Analysis: - standard parametric analysis - censored data; survival analysis -a priori defined stratification (e.g. gender) - a cognitive assessments for covariate analysis (e.g. verbal memory, technology use) right-side graphic: screen shot of video game floor plan; lable, "Data Capture." Description End: So let's take a look at some of the data. Before I do, here, here's some-- some more information. The inclusion criteria we took 18-year-olds, anywhere aged between 18 and 45. I will show you a youth study that we did specifically right after that. Male and female. We documented legal blindness before the age of three. And blindness of ocular cause, regardless of their level of visual acuity or res- residual visual function, they were all blindfolded throughout the study as they- as they played the game. Outcome measures were things of the number of paths that they got correct. The time target. And also what we called creativity points. In other words how well were they able to find their way out the quickest way possible. I'll explain that to you more specifically. Qualitative things like the types of errors they made, things for example like strategies employed, all of that was, was documented, as well, to get a sense of what was happening. Stopping rules. they were given a maximum of six minutes per trial. The reason for this is that we didn't want to have people just indefinitely exploring and then finding their way. So you had six minutes maximum before we went to the next step. And we also had a stopping rule in terms of three successful successive trials to move on to the next one. So if you weren't able to get three in a row at the very, very beginning we stopped the study completely. There was a lot of concern from the Carroll Center that a lot of these kids might get dejected from the performance as well, so we had these stopping rules in there to try to get away from any concerns. And I can tell you that it actually never happened. It was actually quite- quite straightforward. The analysis again just- just some details here in terms of how, how we were able to do that. And just to give you some details: the software-- the nice thing about it- is that it allows us to quantify a lot of things. Here's the path that the individual took. Right? How much time they took in the various parts, all this can be quantified and enters into the spreadsheet so we can break down the path and see areas that they struggled with versus which paths were more challenging than others. [ Slide end: ] [ Slide start: ] Description Start: Title: Transfer of Navigation Skills Learned with AbES to Real World Content: left-side text: Target destination navigation left-side graphic: screen shots of first and second floor floor plans with player path in yellow. left-side text: (Directed Navigator) virtual navigation time: 1 min 42 sec physical navigation time: 1 min 05 sec (Gamer) virtual navigation time: 1 min 24 sec physical navigation time: 1 min 10 sec right-side text: Alternate route navigation right-side graphic: screen shots of first and second floor floor plans for both Gamer and Directed Navigator players; paths traced in yellow. right-side text: Gamer: virtual navigation time: 23 sec physical navigation time: 35 sec Directed Navigator: virtual navigation time: 1 min 15 sec physical navigation time: 1 min 9 sec Description End: So let's take a look at some of the, the quick data before I give you the group analysis. So here are two individuals: one was in the directed navigator group and the other one was in the gamer group. All right? So notice that when we asked them to virtually navigate from the, the lobby of the building all the way through up the stairwell to the second floor to bedroom six, it took them about a minute and 42 seconds to do it. The reason why I chose this path is it's actually the longest path in the building physically. And they took about a minute and 42 seconds to do it. When we take them to the building and ask them to do the same thing, they can do it in a little bit shorter time. Part of that is the physical translation, and the second part is the fact that they're doing the task twice obviously. The thing that's noticeable is notice that the gamers did it equally well in about the same amount of time as well. So whether you learn this as a directed navigator or you learn this as a gamer, you are able to do this. Those individuals in the third arm-- the control group- weren't able to transfer at all, as you might imagine. They got there and we say, "Get to Bedroom Six" and they're like "What's Bedroom Six?" Right? So the context aspect obviously was crucial. None of the individuals in that control group-- that third arm-- were able to do the task. Here's-- here's what's interesting. Once they're at Bedroom Six for example, this particular location, we ask them, "What's the quickest way out of the building?" The gamers always find the quickest way out. The directed navigators just retrace their path, the way that they came in. Right? Which is probably not, not surprising to you. All right? [ Slide end: ] So something tells us that the way that they manipulate the information from the gaming-- learning it through gaming-- versus how they do it through directed navigation is probably different, even though they're very similar on the first task getting from A to B, C to D, how they manipulate that information seems to be very different. This is how we quantified it. [ Slide start: ] Repeat previous slide If you can get the quickest way out-- there was always three ways at least to get out of the building. If you find the shortest route, we give you three points. You find the second shortest route we give you two points. And if you find the longest route that's one. And if you get lost, you can't final your way in the six minutes, you get zero. Pretty simple and we call those "creativity points." Right? So. Other thing just very, very quickly to notice. Notice how they're actually shorelining? Very, very similar to what they actually do in the real world as well. They use very, very much the same strategies in the virtual world that they actually do in the real world, as well. [ Slide end: ] [ Slide start: ] Description Start: Title: Transfer of Navigation Skills Learned with AbES to Real World Content: left-side photo: Girl with white cane and blindfold navigating hallway; thought bubble shows a arrow with left and right turns. left-side text: A. Physical Navigation left-side graphic: bar chart; y-axis label, "Correct Paths Taken (%);" scale, 0 to 120; red (Gamers) and blue (Navigators) bars for Early Blind and Late Blind, roughly equal (90%). right-side photo: Girl with white cane and blindfold navigating hallway; thought bubble shows a straight arrow. left-side text: A. Drop Off left-side graphic: bar chart; y-axis label, "Average Number of Points;" scale, 0 to 3.5; red (Gamers) and blue (Navigators) bars for Early Blind and Late Blind; speaker describes difference in bar sizes. center text: legend; red = Gamers; blue = Navigators; (n=31) *= p<0.05 Description End: Okay. Here's the hard data and there was over 31-- I have the number here- over 31 subjects who participated in this study, again I'm not showing you the control arm group because none of the people were able to do that. [ Slide end: ] I'm going head-to-head comparison of those in a directed navigator arms, versus those in the gaming arms, and I've separated them from early blind to late blind as well. Into two groups as well. Because we wanted to see whether prior visual imagery had somehow an effect on the- on the possible performance as well. And here's the data. So in the early blind group, whether you were able-- whether you learned through gaming in red or navigating through blue, you had almost 90% correct. You could find your route very, very easily in this regard. There was no statistical difference between the two groups. In the case of late blind: very, very similar performance as well. So take home message one: they are able to do this task quite well; almost 90%, anywhere between 80 and 90% performance or correctness trying to find that route, and it's the same whether you were directed navigator or gamer, and it was the same whether early blind versus late blind. The dropoff task- the creativity task, if you will-- is where we saw the biggest difference, right? The gamers always or generally were always able to find the shorter routes. Whereas the directed navigators always choose the longer paths, as well. And this is true whether you were early blind or late blind and this was statistically significant, as well. All right? [ Slide end: ] So we also did a follow‑up study in terms of adolescence because a lot of concerns that we had is that, "Well, if you do the virtual navigation first, you're basically consolidating the path in your mind, and then when I take you to the physical place you're executing on that path. So we re-- redid the study design in a way to try to get specifically at that question. And we did this specifically also in teens, between 14 and 18 years old; and this is how we designed it in this particular case. [ Slide start: ] Description Start: Title: Transfer of Navigation Skills Learned with AbES to Real World Content: center photo: girl with blindfold and headphones at laptop computer center photo: girl with white cane and blindfold navigates hallway left-side text: Adolescent Substudy (14-18 yrs old; all early blind) n=7 left-side graphic: study flow chart; center block, "Enrolled;" center block, "AbES Game Play;" split into two columns; left, "Task 1: direct route finding," right, "Task 2: exit route finding;" left, "Task 2: exit route finding;" right, "Task 1: direct route finding left-side graphic: table with statistical results; data described by speaker. Description End: We enrolled them, they played AbES the video game, and there were two randomized arms. In the first case you did the direct route, then you did the dropoff task. And the second arm you did the dropoff task then you did the route. So there was no carryover effect of what you did on first task versus the other. It was a wash in this. And what we found in this case: very, very similar performance. The direct navigation group, Task One, mean performance was about 70%. Task Two, the dropoff task, mean performance was 97. All right? So the gamers all did very, very well on this. And the other thing too is the mean shortest path was about 71%. So 71% of the time they would typically take the shortest path. Right? [ Slide end: ] So the gaming itself-- whether they-- whatever the task order was, seemed to indeed allow this, this potential benefit of transfer in the navigation task. Couple of other things to think about which was interesting. [ Slide start: ] Description Start: Title: Transfer of Navigation Skills Learned with AbES to Real World Content from previous slide with right-side graphic: A and B graphs showing results; described by speaker Description End: In terms of their performance we noticed that the more jewels they found- in other words, the better they played the game- the better their overall performance was. This is true for Task One and Task Two. So the better you played the game, the better you actually learned and the better you actually transferred into the real world. In contrast, if you look at performance as a function of number of years of O&M skill, there was no association. So it wasn't biased by the fact that these kids may have had more independence or were better at orientation and mobility. We found that there was no significant association between the two. Performance was actually directly correlated to how well you played the game. [ Slide end: ] [ Slide start: ] Description Start: Title: Transfer of Mental Map to the Real World: Summary Content: left-side graphic: screen shot of game floor plan with start and stop labels. right-side text: Results: 1. 85% success rate in finding target destination 2. correlation between navigation success and game play 3. alternate routes found from free exploration 4. Recall: gamers were never told to learn the layout Reference: Description End: Okay. So various results to think about. So, first of all, as I said, it's about an 85% success rate when it comes to just going from one target to another, A to B. There was a correlation between navigation success and game play. Other things that we noticed: alternate routes were always-- or typically-- found when you learned it through gaming as opposed to direct didactic instruction. And recall also the gamers were never told to learn the layout, right? They basically learned the layout for free, which is we simply said, "This is a game, this is how you play it" and they got the map for free, just by interacting with the map. And the argument that I would make to you is those maps were more flexible. The way that they manipulated the information in their mind as a gamer was very, very different in the case of a directed navigator, which is what I'm summarizing here. [ Slide end: ] [ Slide start: ] Description Start: Title: Transfer of Navigation Skills Learned with AbES to Real World Content: left-side text: Significance left-side graphic: cartoon character on a laptop; thought bubble has screen shot of game floor plan. center graphic: block arrow points to the right-side graphic. right-side graphic: cartoon character exits maze with hand held high in victory. left-side graphic: perspective view of a maze with red arrow tracing successful path through maze. left-side text: Directed Navigators - didactic learning - structured -route knowledge left-side graphic: perspective view of a maze with red arrow tracing diagonal through maze. right-side text: Gamers - exploratory learning - self discovery and pace - survey knowledge. Description End: But in both cases they're able to form the map in their mind, they're able to transfer that to a real world setting, but what I would submit to you is that in the case of directed navigators, they were somewhat-- constrained if you will because of the didactic learning. They could only use the information that they were taught. Right? So structure-- basically what you would call "root knowledge" all right, in terms of O&M. Whereas the gamers, exploratory learning, self-discovery and pace allowed a certain cognitive flexibility that they didn't have in the case of the directed navigators. You might want to call this "survey knowledge," for example, in the case of O&M. Right? So big difference in terms of how they were able to perform even though similar performances in some aspects, very different performance differences in, in other aspects as well. [ Slide end: ] [ Slide start: ] Description Start: Title: What are the Neural Correlates of Virtual Navigation? Content: center graphic: 3D drawing of a head, viewed from above; maze etched in the head. Description End: So. How does the brain do this? Right? At the end of the day we stick everybody into the scanner, right, that's what we do. [Laughs] [ Laughter ]. Right? It's all about getting to the scanner. [ Laughter ] So let's talk about the neuroscience behind this. That was the behavioral aspect. How do they do this, now? So let me tell you a little background behind navigation and so on and how the brain does this. A lot of the first initial work about navigation and finding your way, interestingly enough, was done studying London taxi drivers. If you've ever been to London you know that's just a terrible place to drive. Imagine being a taxi driver. And if you want to be licensed in the city of London to drive a taxi you have to do something what's called "The Knowledge" where you spend two years of intensive dri-- intensive teaching, I should say-- intensive learning memorizing the map of London, and they go through very, very interesting exercises where they have to close their eyes and mentally imagine the route that they would take. So they close their eyes and the instructor will say, "Okay, you pick up a fare at Piccadilly, how do you bring them to Buckingham Palace?" So they're like, "This, or this, I turn this right, go three, three streets, turn right, left, so on." And they memorize that map through this mental exercise all the time. They go two years of this before they actually get the license to drive. Very, very clever. [ Slide start: ] Description Start: Title: The London Taxi Driver Study; Eleanor Maguire and colleagues, University College London Content: photo of traditional London taxi; drawings and MRI images of brain with labels pointing to areas of activity, described by speaker; screen shot from video game Crazy Taxi in London. Description End: It was-- Eleanor McGuire who studied this in London and her students and they had a very, very clever idea is they decided to take these taxi drivers and look at their hippocampus, which you know is the part of the brain responsible for memory and spatial learning. And what they found is, they were-- not only was it larger in these London taxi drivers, it was actually correlated with the number of years that they drove the taxi. So there is structural evidence that their part of the brain physically changed. They then compared that to Londoners who drive in London, but weren't taxi drivers, and there was absolutely no change over time. So their hippocampus was bigger than an aged-matched Londoner who didn't drive a taxi, and it got bigger the longer you drove a taxi, as well. So, interesting associative evidence between the two. The other thing that they figured out was the network of the part of the brains -- or-- that were--- or the part of the brain that was responsible for it. So interesting piece is the parietal cortex, you probably know again responsible for spatial processing. Hippocampus, as I mentioned, in terms of memory and and root learning. The frontal cortex involved with executive decisions, right? And of course the visual cortex because you're-- you have to-- you have to use visual information around you and integrate that. And how did they figure all that out? Using video games. Right? So, this is Crazy Taxi in London. They took London taxi drivers and asked them to play the video game in the scanner, right? And they identified all these areas that I mentioned: you know, parietal cortex, visual cortex, frontal areas, and so on. So here's video games showing up again allowing us to figure out what a taxi driver's brain looks like in terms of the map. [ Slide end: ] [ Slide start: ] Description Start: Title: AbES and Functional Neuroimaging Content: center text: Virtual Navigation in a scanner. center photo: person lays in MRI; mirror above their head; floor map of video game on the wall in the background; lables, "control keys," "reflecting mirror," "AbES projected on a screen," "subject wearing headphones." Description End: So with that in mind, we went back to our gamers and this is how we did it. So this is a sighted control in the f-M-R-I scanner. He's looking through a mirror here going through the video game like this. He's using headphones, looking through the mirror. He can see the screen projected behind him. He's using a series of keys to move left and right and so on, exactly the same way that you would -- with a keyboard of a laptop and he's doing this visually. We then bring in our blind participants and doing exactly the same thing. Obviously the monitor isn't on. And we compare the two in terms of how they're able to do that. [ Slide end: ] [ Slide start: ] Description Start: Title: AbES and Functional Neuroimaging Content: left-side graphic: label, "Sighted (blindfolded);" colorized image of the brain with active areas in yellow and red; described by speaker. right-side graphic: label, "Early Blind;" colorized image of the brain with active areas in yellow and red; described by speaker. Description End: What we found is sure enough a network of activation very, very similar to what we saw in the taxi drivers as well. So visual areas. Auditory cortex is active, right, because they're hearing the sounds. Frontal areas very, very important for executive decisions. We also saw activation in the motor areas-- this is because they're using the keys moving around. And sure enough, activation of visual cortex, as well as a parahippocampus which you see right now. So, all the areas that were identified in the London taxi study, we found the same network in our sighted participants, as well. What do you think the brain looks like in our ocular blind participants? The same. Yeah. Exactly the same. Same areas. Auditory, motor, frontal, visual cortex, parahippocampus. Again, the connectivity is all there. They're using the same network, even though they're not necessarily using that visual information same way. So they're driving the same system through another, another portal, if you will. [ Slide end: ] [ Slide start: ] Description Start: Title: Task Dependent Activation Content: three graphics, each with left and right profile images of the brain with active areas in yellow and red; described by speaker. left-side graphic: "instructions vs rest" right-side graphic: "control vs rest" center graphic: "navigation vs rest" Description End: Let's look at this a little bit more systematically, all right? So for example, here all this brain activation and so on is it indeed related to, to the actual video game? Here's a blind individual. We asked them to just simply listen to the instructions. "Don't play the game, don't move, don't do anything, just listen to the instructions" and we have activation in auditory cortex. We also have activation in sensory-motor areas, because they're mentally imagining doing that motion. Here's the same individual just randomly walking. So they're listening to the cues, and we just say, "Walk in a circle. Don't go anywhere." And again, activation in auditory cortex and sensory-motor areas, as well. Now we ask the individual, "We want you to walk from A to B, C to D" in a goal-directed fashion and that's where you see everything light up. Right? So it's the engagement that does this. Right? Again, going back to that earlier question: How do you turn the brain on? They could do something really hard. [Laughs] It really likes that. That's how you create all of that engagement. Other things that were kind of interesting. [ Slide end: ] [ Slide start: ] Description Start: Title: What is Activation Related to? Content: Four profile images of the brain with active areas in yellow and red; described by speaker.Description End: We started looking at all of our participants, one by one. And we saw a really, really wide variability in terms of activation. We saw some people really, really locked in. All sorts of activation everywhere. Another individual less so. Right? Then we saw other individuals that had really, really strong visual cortex activation, other individuals basically nothing. Or were using other areas of the brain were active. So we wanted to make sense of this. Why were everybody, who's playing this game, using different parts of their brain? [ Slide end: ] Right? What was behind this? Is there somehow we can take these individuals and associate that with their performance in terms of the game, and how they were actually using information? So the way we decided to do this, tried to associate brain activity with behavior, is we used the rationale-- and this gets back to the earlier question about R-O-P that I promised I would allude to. [ Slide start: ] Description Start: Content: Development of a self-report measure of environmental spatial ability; Mary Hegarty, Anthony E. Richardson, Daniel R. Montello, Kristin Lovelace, Ilavanil Subbiah 1. I am very good at giving directions. 8. I have trouble understanding directions. 14 I can usually remember a new route after I have traveled it only once. center graphic: Table with six columns and nine rows detailing results of survey; described by speaker. Description End: So we used something called the "Development of Self‑Report Measure of Environmental Spatial Ability." This was done by Mary Hegarty, and she developed a scale that was first developed for sighted individuals, translated for the blind, in terms of trying to figure out how independent an individual was in terms of their orientation, mobility and navigation skills. So it's questions like "I'm very good at giving directions" on a scale of say one to eight. Eight being very good, one being very poor. "I have trouble understanding directions." Right? So notice the negativity on this one, so we ask it in both ways to make sure that for-- we don't get biased by one polarity versus the other. "I can usually remember a new route after I have traveled it only once," scale of one to eight. And then it goes through analysis and it gives you an independent score. The higher your number, the more theoretically independent you are and more confident you are in terms of your travel. All right? Here are our nine ind-- participants in the f-M-R-I study ranked order by their independence score from one to nine. And what do you notice? Retinopathy prematurity is in the lower half. Right? So just a first piece about this aspect of whether or not R-O-P is somehow related with spatial-- spacial aspects. But I'll get back to that during the question period. The main thing that I thought was quite interesting is that their independent score was really interestingly related with their primary mobility. So the top independence for long cane users-- middle scores were using guide dogs, and the lower scores were all people, for example, using the ride program or-- or having a driver taking them around and so on. And remember we don't ask this specifically. This actually came out as a function of the questionnaire that we asked. So they were rank ordered and it seemed to parallel very, very much their primary mobility aids as well. [ Slide end: ] So we had good, good sense that we were able to rank order these individuals in a real world setting. When we do that, we take the scale, we put that into an equation based on brain activity, and we asked the software to tell us what part of the brain correlates the best with their independence? [ Slide start: ] Description Start: Title: AbES and Functional Neuroimaging Content: label, "Navigation vs Rest ("Execution") left-side graphic: left and right profile fMRI scans of the brain with active areas in yellow and red, label, "Temporal-parietal Junction (TPJ)"; described by speaker. right-side graphic: Graph of TPJ activation (y-axis) and Sense of Direction (x-axis); described by speaker. Description End: And the part of the brain was this area here, called the "temporal parietal junction" or "T-P-J." Right? So there's the correlation analysis, brain activation as a function of their independent score. And of all of the parts of the brain this is one that was intimately related to their independence level. [ Slide end: ] And the reason why I think this is interesting is T-P-J is the part of the brain that's normally active when we tell ourselves stories. So it's kind of interesting that those individuals who are the most independent in terms of navigation are probably the people who somehow, necessarily, can rehearse that story in their mind of where they're going. [ Slide start: ] Repeat previous slide It wasn't necessarily the visual cortex, it wasn't necessarily frontal cortex. The part of the brain that kind of sits in the middle of everything-- parietal, visual, temporal and frontal. The Nexus, if you will, of all, of all these brain areas, that was the part of the brain most correlated with it. [ Slide end: ] [ Slide start: ] Description Start: Title: Future Work Content: center photo: aerial photo of the Carroll Center for the Blind campus with buildings outlined in yellow; cartoon character with sword and shield with stylized title, "Audio Zelda;" drawing of someone loading a dvd/cd into a computer. Description End: Okay. Where are we heading now? So we started very, very simply with this idea, we took one building, tried to map it, and tried to turn to it. And tried to get a sense of the neuroscience and all of these aspects. Our goal now is to map out the entire campus, as you might imagine. You can imagine going from one building to another to another to another. And we call this sort of Audio Zelda, where you find the key in one building which gives you the map to another building which forces you to find the other building to, again, sort of engage them and map out the whole entire campus. You can also think of this might be something that you can put on C-D, or maybe downloadable from space. You-- or from cloud space, I should say. And if you have a client coming to the Carroll Center they can go and play the game on their own time and once they arrive at Carroll they have a-- already a good idea of what the layout is of-- of the campus, that's our goal right now in a particular study. And we are working towards that. [ Slide end: ] [ Slide start: ] Description Start: Content: Logo for Unity software; multiple screen shots of video game environments, indoor & outdoor; photo of teenage boy with blindfold and headphones playing video game with hand-held controller. Description End: We've changed the platform, we're using something called Unity, which is a very, very simple way to- to- to program virtual environments. The nice thing about it is that you can use virtually any platform you want: Android, Mac, Playstation, however they want to interact with the game, they can use whatever interface they want. So you build it once, play it everywhere, sort of thing. So very, very fast. Allows us to create these environments. Just showing here we call it Haganow- for haptic-audio game application because we're adding tactile components to it as well. Here's the indoor environment, there's the outdoor environment, here's a blind individual you can see interacting with it. So using the audio as I mentioned before and also using the rumble feature of the Xbox controller as well. So when they hit an obstacle they get the feedback and the- and the- and, and as they strafe the frequency changes as well, so there's tactile feedback as well as audio. You also notice this device here, it's called the Falcon- the Novint Falcon. This is a force-feedback device, so they can also use that to knock on the door, to use it also as a virtual cane. So all sorts of immersion between the audio as well as the haptic-tactile as well to give them that sense of the indoor and outdoor environment. And this is in the works right now. [ Slide end: ] [ Slide start: ] Description Start: Title: MovaWii Content: left-side graphic: drawings showing 360 degree functions of handheld remote; photos of students with remote walking indoors and outdoors; screen shots of maps of a park with red line showing user's path. Description End: Other things we've played with: The Wii-mote as I mentioned is an interesting way to use that as a virtual cane, so getting that rumble feature. The problem with the Wii-mote is that if you hit an obstacle you can still put your arm through it. So it starts vibrating. So the nice thing about the- the- the Falcon is that it actually gives you that force feedback. The Wii‑mote is just an alarm and doesn't give you that same sort of tactile immersion. But nonetheless interesting-- this was work that we did in Chile. Child learned the layout of a- of a park. We take the child there, and we can track them, and using that sort of sense they build the map of the layout of the park using- using the Wii‑mote as one way to do that. [ Slide end: ] [ Slide start: ] Description Start: Title: Audiopolis Content: center photo: woman uses a handheld phone and computer keyboard; right-side graphic: map of college campus left-side photo: boy using keyboard & audio speakers to play game; label, "Interact with virtual world" center photo: boy stands at table with red cloth and blocks of different sizes and shapes; label, "Reconstrction" right-side photo: blocks on red cloth resemble graphical map above; ; label, "Final Representation" Description End: Audiopolis- another interesting way. This is a ficti-- a fictional environment. Same sort of strategy. We have the child play again, audio as well as the Wii‑mote. the goal here is to chase a thief that's stolen in this virtual city. You have to find the thief and the-- every building you find leaves cues for the next building you have to find. And you kind of go through in a structured fashion. At the end we give you just the blocks and you have to rebuild the environment that you worked with, and again kind of get a sense of the child's spatial skills, how they put the environment together. We've seen, for example, a lot of kids flip it, for example. They know the very short linear relationship between buildings, but globally they have distortion. So it allows us to kind of diagnose what aspect of their spatial representation seems to be impaired, if at all. [ Slide end: ] [ Slide start: ] Description Start: Title: Massachusetts Bay Transit Authority Content: center graphic: 3D drawing of subway station; center photo: subway station Description End: Other things that we've done. We now embarked with the Massachusetts Transit-- Bay Transit Authority- the M-B-T-A- this runs the subway and the bus system in Boston. If you're from Boston you might recognize this. This is Park Street Station. And we have a situation now where we've modeled Park Street Station in our virtual environment hoping this could be a similar strategy, outside of the Carroll Center. Now using this in public spaces as well. We can use this as an offline survey, you can learn how to explore the station before you go. You could maybe use this as an online system, as well; get information when you're in the station. And it's also a way of tracking. All right? So you can use this as a way of "what are the most common exits that people use?" for example, and you put that into a pool of data as well. [ Slide end: ] [ Slide start: ] Description Start: Title: StripMapp Content: left-side photo: Antonio Grimace drawing right-side graphic: four screen shots from smart phone stip map; described by speaker. Description End: Antonio Grimace is a- a student in my lab - he has Lebers. And a very, very proficient traveler in the bus-- in the Boston subway system and he has this idea- which is very, very intriguing-- of developing a strip map. And you're all familiar what a strip map is, if you take the subway. It's basically a linear representation of all the stations and the sequence, right, where the connections are and so on. So he's developing an app, called Strip Mapp, for exactly that purpose, for the Boston metro system, subway system. And it looks a little something like this. First thing you do, taking advantage of the tactile interface, the- the gestures as well as the audio that you get from your iPhone. You can ask, for example, what direction you want to head in. How do I get from one station to another or find the best route between two stations? You can choose the line that you want, again just scrolling through in a strip map fashion, and it will calculate the optimal route for you. And then you get all sorts of feedback, for example, crowd-sourcing tips that people put on board. You know, "this is a good place to get a coffee" or "this is a particularly complicated place." Live schedule. We get an immediate feed from the M-B-T-A, because of our association, of when the trains are coming, when the subway's coming, if there's any delays- they have it immediately on their phone. Other accessibility services like you know "Call an Uber", "Call a taxi" and so on. So this is in the works, as well. [ Slide end: ] So again: original idea, neuroscience, now trying to translate that into real world applications that I think people-- that people can use. [ Slide start: ] Description Start: Title: Other Software: University of Chile, Department of Computer Science Content: IDAC logo; "Inclusion digital por aprender ciencias;" left-side photo: students with headphones at computer stations center graphic: flow chart of computer game right-side photo: felt body outline with removable organs right-side photo: tactual graphic of the brain with labels in color, large print and braille. Description End: And the last example I'll give you I think a very, very interesting one. This isn't a project that I was involved with. This is my collaborators back in Chile. And this is a project called I-DAC, which is "Inclusion Digital por Aprender Ciencias." So this was a project that they used-- they used video games to teach basic anatomy and biology for blind students. In Chile-- much like here in the United States-- a lot of the kids are mainstreamed, they spend a lot of time in the public school system. And what they do, is they've invented a game where a blind individual has to play with two or three sighted classmates, and the idea is to explore the world through these various rooms, and as they go through the world they learn the anatomy, together. The sighted kids see each other, right, as they move through. And the blind child is using the audio cues to navigate with them. So they're playing the game together. So it's a combination of teamwork, as well as concrete tools as well -- various models and things which they use together. Notice that they're visually labeled, as well as braille. Also all the material is exactly the same, the reading material visually versus the tactile braille material is identical. Idea is to integrate and force the teamwork, have the kids working together. Very, very interesting results. This is a long‑term study with the Ministry of Education, that they're doing. So I think a very, very clever idea; very interesting approach to engage them. [ Slide end: ] [ Slide start: ] Description Start: Title: Content: Reference: Description End: So final slides. A great, great, great quote that I love. "You can discover more about a person in an hour of play than a year of conversation." Turns out Plato said that. [ Slide end: ] I think that's a really, really interesting idea. The fact that play somehow brings out our better nature, if you will, I think is a very interesting one and an intriguing one. We solve problems in ways that we don't normally solve in the-- in the real world. [ Slide start: ] Description Start: Title: "You can discover more about a person in an hour of play than in a year of conversation." - Plato Content: center photo: profile view; guide dog sits at the face of a rock wall looking up a dangling climbing rope. Reference: National Sports Center for the Disabled; www.nscd.org Description End: Great picture that I love. This is another one that's hanging in my-- in my office in the National Sports Center for the Disabled. You see this guide dog looking up at his master- presumably climbing this wall. I love this picture because we have this sense that individuals are only as good as their technology. Right? [ Slide end: ] I think the idea is to go beyond that. I think the idea is to have-- create independence, create confidence, create a level -- of function beyond the tools that are available. And that's what I think what this- what this picture symbolizes. And the last one that I want to share with you to close out, straight out of- out of Texas folklore, which I think is very, very impression-- or sorry, let me- let me thank a couple of individuals first. [ Slide start: ] Description Start: Title: Thank you! Content: Photos of Dr. Jaime Sanchez, University of Chile, Santiago; Erin Connors, Research Assistant; Mark Halko, Carroll Center for the Blind Description End: Jaime Sanchez, as I said, who-- professor at University of Chile. Erin Connors was a research assistant working on this project. Mark Halko did the f-M-R-I work, and of course this- the Carroll Center for Blind where we did a lot of the work. [ Slide end: ] And now my final slide that I want to share with you- again, I think very, very-- very appropriate and right out of Texas folklore. [ Slide start: ] Description Start: Title: CLEAR EYES, FULL HEARTS, CAN'T LOSE Content: night photo of illuminated high school football field Description End: Love this: "Clear eyes, full hearts, can't lose." This-- every time I work with a blind child this is what I think about. It's exactly that. Right? [ Slide end: ] There's so much more to see than the health of the eyes and also the health of the brain. And I think you can experience the world in so many different ways. And I think that's our goal. And if we do that-- keep our hearts full-- can't lose. Right? So again, thank you very much for... [Applause]