Articles
Digital Synesthesia
Want to see with your tongue? Boing Boing's David Pescovitz looks at technology that blurs the boundaries between our five senses. What if you...
09.02.09
Want to see with your tongue? Boing Boing's David Pescovitz looks at technology that blurs the boundaries between our five senses.
What if you could see with your skin? Or taste what you see? While those kinds of experiences might suggest a mental disorder, or an acid trip, the ability to substitute your senses by choice is on the horizon. A confluence of new technologies are leading to a kind of digital synesthesia.
Synesthesia, of course, is the fascinating neurological phenomenon whereby stimulation of one sense involuntarily triggers another sensory pathway. A synesthete might taste sounds or hear colors. But the ability to reroute the senses could dramatically help blind individuals, for example, or restore the sense of touch to amputees wearing prosthetic limbs.
At Institute for the Future, where I'm a researcher, my colleagues and I have spent the last few months exploring the notion that "everything is programmable," or will be soon. The idea is that emerging technologies-from pervasive computers to synthetic biology-are making it possible to program our bodies and our worlds to desired specifications. Increasingly, we are looking at the entire world through a computational lens. As part of that research, we've been collecting "signals"-events, developments, articles, scientific publications-that taken together, give indications of key trends. We've entered these in our public Signtific signals database and tagged them based on their subject matter. I've found many research efforts suggesting how we may reprogram our senses in the future.
For example, there are the "Flavor Tripping" parties fueled by Synsepalum dulcificum, aka "Miracle Fuit," the West African berry that temporarily reprograms your taste buds to make anything sour or bitter taste perfectly sweet. And there's the story of Daniel Kish, the blind psychologist who, by clicking his tongue, uses echolocation to "see." In the realm of digital synesthesia, numerous projects are attempting to leverage tactile feedback in the form of clothing outfitted with tiny vibrators. Instead of picking up your phone to read a text message, you might feel the words spelled on your back.
The late Paul Bach-y-Rita could be considered the father of all technology used to reprogram the human senses. In 1963, Bach-y-Rita developed a "Tactile to Visual Sensory Substitution" device. It converted images from a camera to tactile sensations that a blind person could feel on his or her back. Bach-y-Rita's research was all based on the notion of "sensory substitution." The brain, he argued, was not hardwired and that a working sense, say touch, could be used to replace a failing one, e.g. vision. His ideas around the plasticity of the human brain were very controversial at the time but widely accepted today. He continued his research on sensory substitution technology at the University of Wisconsin-Madison, and his company Wicab, until his death in 2007.
"I can connect anything to anything," Bach-y-Rita said in a profile in The Telegraph shortly before his death:
We see with our brains, not with our eyes. When a blind man uses a cane he sweeps it back and forth, and has only one point, the tip, feeding him information through the skin receptors in the hand. Yet this sweeping allows him to sort out where the doorjamb is, or the chair, or distinguish a foot when he hits it, because it will give a little. Then he uses this information to guide himself to the chair to sit down. Though his hand sensors are where he gets the information and where the cane "interfaces" with him, what he perceives is not the cane's pressure on his hand but the layout of the room: chairs, walls, feet, the three-dimensional space. The receptor surface in the hand becomes merely a relay for information, a data port.
In the latest incarnation of Bach-y-Rita's work, the data port is the tongue. The company he co-founded, Wicab, has developed a visual prosthetic for the blind that converts images from a video camera into tactile sensations on the tongue. The system, called BrainPort, pairs a head-mounted digital video camera with a postage stamp-size electrode array that sits on the tongue. A small computer translates the visual information into a pattern that is "displayed" on the tongue.
"The tactile image is created by presenting white pixels from the camera as strong stimulation, black pixels as no stimulation, and gray levels as medium levels of stimulation, with the ability to invert contrast when appropriate," reads the company's website. "Users often report the sensation as pictures that are painted on the tongue with champagne bubbles."
The BrainPort is not yet FDA approved, but clinical studies have been quite exciting. During trials, blind test subjects had their brains scanned while using the device. Interestingly, even though the device provides tactile sensation, visual regions of the brain were activated.
Seeing with your tongue may seem unusual, but arguably not as weird as "skin vision." A researcher at Tel Aviv University suggests that humans might be able to "see" with their skin. Engineering professor Leonid Yaroslavsky hopes that through biomimicry, new kinds of imaging technology might be developed that obviates traditional optics. Yaroslavsky presented his theories on the subject in a scientific book titled Advances in Information Optics and Photonics. From an American Friends of Tel Aviv University press release:
Skin vision is not uncommon in nature. Plants orient themselves to light, and some animals-such as pit vipers, who use infrared vision, and reptiles, who possess skin sensors-can "see" without the use of eyes. Skin vision in humans is likely a natural atavistic ability involving light-sensitive cells in our skin connected to neuro-machinery in the body and in the brain, explains Prof. Yaroslavsky.
While the first people to reprogram their senses are likely to be people with a sense that has failed them, the technology will likely trickle down. Eventually, the hard lines between our five senses may be blurred. And in a world where everything is programmable, five may be a choice, not a limit.
Do you see what I'm saying?
David Pescovitz is co-editor of Boing Boing, a research director at Institute for the Future, and editor-at-large of MAKE.
Keep ReadingShow less
