Read our COVID-19 research and news.

Computer Program Allows the Blind to 'See' With Sound

A man blind since birth is taking up a surprising new hobby: photography. His newfound passion is thanks to a system that turns images into sequences of sound. The technology not only gives “sight” to the blind, but also challenges the way neurologists think the brain is organized.

In 1992, Dutch engineer Peter Meijer created vOICe, an algorithm that converts simple grayscale images into musical soundscapes. (The capitalized middle letters sound out “Oh, I see!”). The system scans images from left to right, converting shapes in the image into sound as it sweeps, with higher positions in the image corresponding to higher sound frequencies. For instance, a diagonal line stretching upward from left to right becomes a series of ascending musical notes. While more complicated images, such as a person sitting on a lawn chair, at first seem like garbled noise, with enough training users can learn to “hear” everyday scenes.

In 2007, neuroscientist Amir Amedi and his colleagues at the Hebrew University of Jerusalem began training subjects who were born blind to use vOICe. Despite having no visual reference points, after just 70 hours of training, the individuals went from “hearing” simple dots and lines to “seeing” whole images such as faces and street corners composed of 4500 pixels. (For comparison, Nintendo’s Mario was made up of just 192 pixels in his first video game appearance.) By attaching a head-mounted camera to a computer and headphones, the blind users were even able to navigate around a room by the sound cues alone. Every few steps the system snaps a photo and converts it into sound, giving the users their bearings as they traverse tables and trashcans. One patient even took up photography, using the head-mounted system to frame his snapshots.

The training program also devoted 10 hours to recognizing human silhouettes represented by sound. By the end of the training, participants could detect the exact posture of the person represented by the soundscape and replicate the pose, the team reports online today in Current Biology.

When the researchers mapped the brain activity of the participants, they found something astonishing. The generally accepted model of the brain contains regions devoted to each sense, such as the sight-centric visual cortex. Researchers had long believed that if those regions aren’t used for their intended sense, they are repurposed for other uses; for example, the visual cortex of someone blind from birth could be used to help boost her hearing. But Amedi and his colleagues found that the area of the visual cortex responsible for recognizing body shapes in sighted people—called the extrastriate body area—lit up with activity in the study participants when they were interpreting the human silhouettes.

Amedi says the traditional sensory-organized brain model can’t explain this activity; after all, the subjects only heard the information, and scientists believed that the body-recognizing area shouldn’t have fully developed without visual experiences during development. Neuroscientist Ella Striem-Amit of Harvard University, who co-authored the paper, thinks it’s time for a new model. “The brain, it turns out, is a task machine, not a sensory machine,” she says. “You get areas that process body shapes with whatever input you give them—the visual cortex doesn’t just process visual information.”

Ione Fine, a neuroscientist at the University of Washington, Seattle, who studies brain changes and did not work on the project, says Amedi’s work is the best evidence yet for functional constancy—the idea that areas of the brain do the same job even with different kinds of input. "The idea that the organization of blind people’s brains is a direct analog to the organization of sighted people’s brains is an extreme one—it has an elegance you rarely actually see in practice," she says. "If this hypothesis is true, and this is strong evidence that it is, it means we have a deep insight into the brain." In an alternative task-oriented brain model, parts of the brain responsible for similar tasks—such as speech, reading, and language—would be closely linked together.

Amedi's team recently released a successor to vOICe, called EyeMusic, as a free iPhone app. The new algorithm produces more pleasant tones and can even provide color information. In the above video, a man blind since birth uses EyeMusic to “see” drawn faces. Amedi hopes EyeMusic will help blind users gain more independence and improve their quality of life.

(Video credit: Amedi Lab)