Digital Synesthesia

Want to see with your tongue? Boing Boing's David Pescovitz looks at technology that blurs the boundaries between our five senses. What if you...

Want to see with your tongue? Boing Boing's David Pescovitz looks at technology that blurs the boundaries between our five senses.

What if you could see with your skin? Or taste what you see? While those kinds of experiences might suggest a mental disorder, or an acid trip, the ability to substitute your senses by choice is on the horizon. A confluence of new technologies are leading to a kind of digital synesthesia.

Synesthesia, of course, is the fascinating neurological phenomenon whereby stimulation of one sense involuntarily triggers another sensory pathway. A synesthete might taste sounds or hear colors. But the ability to reroute the senses could dramatically help blind individuals, for example, or restore the sense of touch to amputees wearing prosthetic limbs.

At Institute for the Future, where I'm a researcher, my colleagues and I have spent the last few months exploring the notion that "everything is programmable," or will be soon. The idea is that emerging technologies-from pervasive computers to synthetic biology-are making it possible to program our bodies and our worlds to desired specifications. Increasingly, we are looking at the entire world through a computational lens. As part of that research, we've been collecting "signals"-events, developments, articles, scientific publications-that taken together, give indications of key trends. We've entered these in our public Signtific signals database and tagged them based on their subject matter. I've found many research efforts suggesting how we may reprogram our senses in the future.

For example, there are the "Flavor Tripping" parties fueled by Synsepalum dulcificum, aka "Miracle Fuit," the West African berry that temporarily reprograms your taste buds to make anything sour or bitter taste perfectly sweet. And there's the story of Daniel Kish, the blind psychologist who, by clicking his tongue, uses echolocation to "see." In the realm of digital synesthesia, numerous projects are attempting to leverage tactile feedback in the form of clothing outfitted with tiny vibrators. Instead of picking up your phone to read a text message, you might feel the words spelled on your back.

The late Paul Bach-y-Rita could be considered the father of all technology used to reprogram the human senses. In 1963, Bach-y-Rita developed a "Tactile to Visual Sensory Substitution" device. It converted images from a camera to tactile sensations that a blind person could feel on his or her back. Bach-y-Rita's research was all based on the notion of "sensory substitution." The brain, he argued, was not hardwired and that a working sense, say touch, could be used to replace a failing one, e.g. vision. His ideas around the plasticity of the human brain were very controversial at the time but widely accepted today. He continued his research on sensory substitution technology at the University of Wisconsin-Madison, and his company Wicab, until his death in 2007.

"I can connect anything to anything," Bach-y-Rita said in a profile in The Telegraph shortly before his death:

We see with our brains, not with our eyes. When a blind man uses a cane he sweeps it back and forth, and has only one point, the tip, feeding him information through the skin receptors in the hand. Yet this sweeping allows him to sort out where the doorjamb is, or the chair, or distinguish a foot when he hits it, because it will give a little. Then he uses this information to guide himself to the chair to sit down. Though his hand sensors are where he gets the information and where the cane "interfaces" with him, what he perceives is not the cane's pressure on his hand but the layout of the room: chairs, walls, feet, the three-dimensional space. The receptor surface in the hand becomes merely a relay for information, a data port.

In the latest incarnation of Bach-y-Rita's work, the data port is the tongue. The company he co-founded, Wicab, has developed a visual prosthetic for the blind that converts images from a video camera into tactile sensations on the tongue. The system, called BrainPort, pairs a head-mounted digital video camera with a postage stamp-size electrode array that sits on the tongue. A small computer translates the visual information into a pattern that is "displayed" on the tongue.

"The tactile image is created by presenting white pixels from the camera as strong stimulation, black pixels as no stimulation, and gray levels as medium levels of stimulation, with the ability to invert contrast when appropriate," reads the company's website. "Users often report the sensation as pictures that are painted on the tongue with champagne bubbles."

The BrainPort is not yet FDA approved, but clinical studies have been quite exciting. During trials, blind test subjects had their brains scanned while using the device. Interestingly, even though the device provides tactile sensation, visual regions of the brain were activated.

Seeing with your tongue may seem unusual, but arguably not as weird as "skin vision." A researcher at Tel Aviv University suggests that humans might be able to "see" with their skin. Engineering professor Leonid Yaroslavsky hopes that through biomimicry, new kinds of imaging technology might be developed that obviates traditional optics. Yaroslavsky presented his theories on the subject in a scientific book titled Advances in Information Optics and Photonics. From an American Friends of Tel Aviv University press release:

Skin vision is not uncommon in nature. Plants orient themselves to light, and some animals-such as pit vipers, who use infrared vision, and reptiles, who possess skin sensors-can "see" without the use of eyes. Skin vision in humans is likely a natural atavistic ability involving light-sensitive cells in our skin connected to neuro-machinery in the body and in the brain, explains Prof. Yaroslavsky.

While the first people to reprogram their senses are likely to be people with a sense that has failed them, the technology will likely trickle down. Eventually, the hard lines between our five senses may be blurred. And in a world where everything is programmable, five may be a choice, not a limit.

Do you see what I'm saying?

David Pescovitz is co-editor of Boing Boing, a research director at Institute for the Future, and editor-at-large of MAKE.

Julian Meehan

Young leaders from around the world are gathering at the United Nations Headquarters in New York Saturday to address arguably the most urgent issue of our time. The Youth Climate Summit comes on the heels of an international strike spearheaded by Greta Thunberg, the 16-year-old climate activist from Sweden, who arrived in New York via emissions-free sailboat earlier this month.

Translated from Swedish, "berg" means "mountain," so it may feel fated that a young woman with Viking blood in her veins and summit in her name would be at the helm. But let's go out on a limb and presume Thunberg, in keeping with most activists, would chafe at the notion of pre-ordained "destiny," and rightly so. Destiny is passive — it happens to you. It's also egomaniacal. Change, on the other hand, is active; you have to fight. And it is humble. "We need to get angry and understand what is at stake," Thunberg declared. "And then we need to transform that anger into action."

This new generation of activists' most pernicious enemy is denial. The people in charge — complacent politicians and corporation heads who grossly benefit from maintaining the status quo — are buffered from real-life consequences of climate change. But millions of people don't share that privilege. For them, climate change isn't an abstract concept, but a daily state of emergency, whether it comes in the form of "prolonged drought in sub-Saharan Africa…devastating tropical storms sweeping across Southeast Asia, the Caribbean and the Pacific…[or] heatwaves and wildfires," as Amnesty International reportsare all too real problems people are facing on a regular basis.

RELATED: Greta Thunberg urges people to turn to nature to combat climate change

Keep Reading Show less
The Planet

Millions of people in over 150 countries across the globe marched for lawmakers and corporations to take action to help stop climate change on Friday, September 20.

The Climate Strikes were organized by children around the world as an extension of the of the "Fridays for Future" campaign. Students have been walking out of classrooms on Fridays to speak out about political inaction surrounding the climate crisis.

"We need to act right now to stop burning fossil fuels and ensure a rapid energy revolution with equity, reparations and climate justice at its heart," organizers say.

There's no doubt the visual images from the marches send a powerful message to those on the ground but especially those watching from around the world. GOOD's own Gabriel Reilich was on the scene for the largest of the Climate Strikes. Here are 18 of the best signs from the Climate Strike march in New York City.

Keep Reading Show less

September 20th marks the beginning of a pivotal push for the future of our planet. The Global Climate Strike will set the stage for the United Nations Climate Action Summit, where more than 60 nations are expected to build upon their commitment to 2015's Paris Agreement for combating climate change.

Millions of people are expected to take part in an estimated 4,000 events across 130 countries.

Keep Reading Show less
The Planet
via Apple

When the iPhone 11 debuted on September 10, it was met with less enthusiasm than the usual iPhone release. A lot of techies are holding off purchasing the latest gadget until Apple releases a phone with 5G technology.

Major US phone carriers have yet to build out the infrastructure necessary to provide a consistent 5G experience, so Apple didn't feel it necessary to integrate the technology into its latest iPhone.

A dramatic new feature on the iPhone 11 Pro is its three camera lenses. The three lenses give users the the original wide, plus ultrawide and telephoto options.

Keep Reading Show less
via I love butter / Flickr

We often dismiss our dreams as nonsensical dispatches from the mind while we're deep asleep. But recent research proves that our dreams can definitely affect our waking lives.

People often dream about their significant others and studies show it actually affects how we behave towads them the next day.

"A lot of people don't pay attention to their dreams and are unaware of the impact they have on their state of mind," said Dylan Selterman, psychology lecturer at the University of Maryland, says according to The Huffington Post. "Now we have evidence that there is this association."

Keep Reading Show less