Digital Synesthesia

Want to see with your tongue? Boing Boing's David Pescovitz looks at technology that blurs the boundaries between our five senses. What if you...

Want to see with your tongue? Boing Boing's David Pescovitz looks at technology that blurs the boundaries between our five senses.

What if you could see with your skin? Or taste what you see? While those kinds of experiences might suggest a mental disorder, or an acid trip, the ability to substitute your senses by choice is on the horizon. A confluence of new technologies are leading to a kind of digital synesthesia.

Synesthesia, of course, is the fascinating neurological phenomenon whereby stimulation of one sense involuntarily triggers another sensory pathway. A synesthete might taste sounds or hear colors. But the ability to reroute the senses could dramatically help blind individuals, for example, or restore the sense of touch to amputees wearing prosthetic limbs.

At Institute for the Future, where I'm a researcher, my colleagues and I have spent the last few months exploring the notion that "everything is programmable," or will be soon. The idea is that emerging technologies-from pervasive computers to synthetic biology-are making it possible to program our bodies and our worlds to desired specifications. Increasingly, we are looking at the entire world through a computational lens. As part of that research, we've been collecting "signals"-events, developments, articles, scientific publications-that taken together, give indications of key trends. We've entered these in our public Signtific signals database and tagged them based on their subject matter. I've found many research efforts suggesting how we may reprogram our senses in the future.

For example, there are the "Flavor Tripping" parties fueled by Synsepalum dulcificum, aka "Miracle Fuit," the West African berry that temporarily reprograms your taste buds to make anything sour or bitter taste perfectly sweet. And there's the story of Daniel Kish, the blind psychologist who, by clicking his tongue, uses echolocation to "see." In the realm of digital synesthesia, numerous projects are attempting to leverage tactile feedback in the form of clothing outfitted with tiny vibrators. Instead of picking up your phone to read a text message, you might feel the words spelled on your back.

The late Paul Bach-y-Rita could be considered the father of all technology used to reprogram the human senses. In 1963, Bach-y-Rita developed a "Tactile to Visual Sensory Substitution" device. It converted images from a camera to tactile sensations that a blind person could feel on his or her back. Bach-y-Rita's research was all based on the notion of "sensory substitution." The brain, he argued, was not hardwired and that a working sense, say touch, could be used to replace a failing one, e.g. vision. His ideas around the plasticity of the human brain were very controversial at the time but widely accepted today. He continued his research on sensory substitution technology at the University of Wisconsin-Madison, and his company Wicab, until his death in 2007.

"I can connect anything to anything," Bach-y-Rita said in a profile in The Telegraph shortly before his death:

We see with our brains, not with our eyes. When a blind man uses a cane he sweeps it back and forth, and has only one point, the tip, feeding him information through the skin receptors in the hand. Yet this sweeping allows him to sort out where the doorjamb is, or the chair, or distinguish a foot when he hits it, because it will give a little. Then he uses this information to guide himself to the chair to sit down. Though his hand sensors are where he gets the information and where the cane "interfaces" with him, what he perceives is not the cane's pressure on his hand but the layout of the room: chairs, walls, feet, the three-dimensional space. The receptor surface in the hand becomes merely a relay for information, a data port.

In the latest incarnation of Bach-y-Rita's work, the data port is the tongue. The company he co-founded, Wicab, has developed a visual prosthetic for the blind that converts images from a video camera into tactile sensations on the tongue. The system, called BrainPort, pairs a head-mounted digital video camera with a postage stamp-size electrode array that sits on the tongue. A small computer translates the visual information into a pattern that is "displayed" on the tongue.

"The tactile image is created by presenting white pixels from the camera as strong stimulation, black pixels as no stimulation, and gray levels as medium levels of stimulation, with the ability to invert contrast when appropriate," reads the company's website. "Users often report the sensation as pictures that are painted on the tongue with champagne bubbles."

The BrainPort is not yet FDA approved, but clinical studies have been quite exciting. During trials, blind test subjects had their brains scanned while using the device. Interestingly, even though the device provides tactile sensation, visual regions of the brain were activated.

Seeing with your tongue may seem unusual, but arguably not as weird as "skin vision." A researcher at Tel Aviv University suggests that humans might be able to "see" with their skin. Engineering professor Leonid Yaroslavsky hopes that through biomimicry, new kinds of imaging technology might be developed that obviates traditional optics. Yaroslavsky presented his theories on the subject in a scientific book titled Advances in Information Optics and Photonics. From an American Friends of Tel Aviv University press release:

Skin vision is not uncommon in nature. Plants orient themselves to light, and some animals-such as pit vipers, who use infrared vision, and reptiles, who possess skin sensors-can "see" without the use of eyes. Skin vision in humans is likely a natural atavistic ability involving light-sensitive cells in our skin connected to neuro-machinery in the body and in the brain, explains Prof. Yaroslavsky.

While the first people to reprogram their senses are likely to be people with a sense that has failed them, the technology will likely trickle down. Eventually, the hard lines between our five senses may be blurred. And in a world where everything is programmable, five may be a choice, not a limit.

Do you see what I'm saying?

David Pescovitz is co-editor of Boing Boing, a research director at Institute for the Future, and editor-at-large of MAKE.


Looking back, the year 1995 seems like such an innocent time. America was in the midst of its longest streak of peace and prosperity. September 11, 2001 was six years away, and the internet didn't seem like much more than a passing fad.

Twenty-four years ago, 18 million U.S. homes had modem-equipped computers, 7 million more than the year before. Most logged in through America Online where they got their email or communicated with random strangers in chat rooms.

According to a Pew Research study that year, only 32% of those who go online say they would miss it "a lot" if no longer available.

Imagine what those poll numbers would look like if the question was asked today.

RELATED: Bill and Melinda Gates had a surprising answer when asked about a 70 percent tax on the wealthiest Americans

"Few see online activities as essential to them, and no single online feature, with the exception of E-Mail, is used with any regularity," the Pew article said. "Consumers have yet to begin purchasing goods and services online, and there is little indication that online news features are changing traditional news consumption patterns."

"Late Night" host David Letterman had Microsoft founder and, at that time the richest man in the world, on his show for an interview in '95 to discuss the "the big new thing."

During the interview Letterman chided Gates about the usefulness of the new technology, comparing it to radio and tape recorders.

Gates seems excited by the internet because it will soon allow people to listen to a baseball game on their computer. To which Letterman smugly replies, "Does radio ring a bell?" to laughter from the crowd.

But Gates presses Letterman saying that the new technology allows you to listen to the game "whenever you want," to which Letterman responds, "Do tape recorders ring a bell?"

Gates then tells Letterman he can keep up with the latest in his favorite hobbies such as cigar smoking or race cars through the internet. Letterman shuts him down saying that he reads about his interests in magazines.

RELATED: Bill Gates has five books he thinks you should read this summer.

The discussion ends with the two laughing over meeting like-minded people in "troubled loner chat room on the internet."

The clip brings to mind a 1994 segment on "The Today Show" where host Bryant Gumbel and Katie Couric have a similar discussion.

"What is internet anyway?" an exasperated Gumball asks. "What do you write to it like mail?"

"It's a computer billboard but it's nationwide and it's several universities all joined together and it's getting bigger and bigger all the time," a producer explains from off-stage.

via The Howard Stern Show / YouTube

Former Secretary of State, first lady, and winner of the popular vote in the 2016 presidential election, Hillary Clinton, sat own for an epic, two-and-a--half hour interview with Howard Stern on his SiriusXM show Wednesday.

She was there to promote "The Book of Gutsy Women," a book about heroic women co-written with her daughter, Chelsea Clinton.

In the far-reaching conversation, Clinton and the self-proclaimed "King of All Media" and, without a doubt, the best interviewer in America discussed everything from Donald Trump's inauguration to her sexuality.

Keep Reading Show less

The healthcare systems in the United States and the United Kingdom couldn't be more different.

The UK's National Health Service is the largest government-run healthcare system in the world and the US's is largest private sector system.

Almost all essential health services in the UK are free, whereas in America cost can vary wildly based on insurance, co pays and what the hospitals and physicians choose to charge.

A medical bill in the US

One of the largest differences is cost. The average person in the UK spends £2,989 ($3915) per year on healthcare (most of which is collected through taxes), whereas the average American spends around $10,739 a year.

So Americans should obviously be getting better care, right? Well, the average life expectancy in the UK is higher and infant mortality rate is lower than that in the US.

RELATED: The World Health Organization declares war on the out of control price of insulin

Plus, in the U.S., only 84% of people are covered by private insurance, Medicare or Medicaid. Sixteen percent of the population are forced to pay out of pocket.

In the UK, everyone is covered unless they are visiting the country or an undocumented resident.

Prescription drugs can cost Americans an arm and a leg, but in the UK, prescriptions or either free or capped at £8.60 ($11.27).

via Wikimedia Commons

The one drawback to the NHS system is responsiveness. In the UK people tend to wait longer for inessential surgeries, doctor's appointments, and in emergency rooms. Whereas, the US is ranked as the most responsive country in the world.

RELATED: Alarmingly high insulin prices are forcing Americans to flock to Canada to buy the drug

The New York Times printed a fair evaluation of the UK's system:

The service is known for its simplicity: It is free at the point of use to anyone who needs it. Paperwork is minimal, and most patients never see a bill. … No one needs to delay medical treatment until he or she can afford it, and virtually everyone is covered. …

According to data from the Organization for Economic Cooperation and Development, the United States spent 17.2 percent of its economic output on health care in 2016, compared with 9.7 percent in Britain. Yet Britain has a higher life expectancy at birth and lower infant mortality.

Citizens in each country have an interesting perspective on each other's healthcare systems. UK citizens think it's inhumane for Americans have to pay through the nose when they're sick or injured. While Americans are skeptical of socialist medicine.

A reporter from Politics Joe hit the streets of London and asked everyday people what they think Americans pay for healthcare and they were completely shocked.