This Device Translates Sign Language In Real Time

Illustration by Tyler Hoehne

Once in high school, Mohamed Elwazer was waiting for his train on an empty metro platform in Egypt when he overheard someone yelling in frustration.

“I looked to see what was going on and I found a police officer and a teenager trying to use his hands to say something,” says Elwazer. The teen was clearly hearing impaired, but “the police officer kept shouting at him and didn’t know what he was trying to say.”

Fourteen years later, Elwazer, a computer systems engineer, remembers this incident as the genesis of his life’s work, KinTrans, a computer system that translates sign language in real time. On that platform, Elwazer realized there was a need for such a device, but it was after watching a group of kids play an Xbox 360 that he figured out how he could create one. He used the same Microsoft Kinect software integrated into the gaming system to build KinTrans. Users stand in front of the camera and motion sensor technology reads the signals, immediately translating them into voice and text.

“We are translating sign language in the most convenient way for the hearing impaired,” says Elwazer, “in real time, as [they] sign, and without electronics in their hands.”

With the fast pace of technology, accessibility devices and apps are more effective and more available than ever for people in the disability community. Developers have been experimenting with wearable products like Acceleglove and Enable Talk, hand gloves that also translate hand signals into sound. But KinTrans is different in that it places the onus on the hearing world to adapt for hearing-impaired people.

Elwazer intends to make KinTrans available to businesses and public institutions, so the hearing-impaired won’t need to carry around any devices. In Dubai, he’s been working with transportation and police authorities on how to integrate KinTrans. They’ve even tested it out at the Dubai International Airport.

There’s also a financial incentive for institutions like these to adopt this kind of technology. Not only will it improve relations with clients who are hearing impaired, it’s also cheaper than employing a full-time interpreter.

“If you calculated the cost of a human interpreter and the machine, our machine is like 90 percent less than the cost of a human interpreter,” says Elwazer. “And the machine is available 24/7.”

KinTrans will also be able to serve people across borders. Sign language evolves as fluidly as spoken language and, over the years, distinct regional signs have emerged. Advocates for the hearing impaired fear that standardizing the language will remove nuances specific to these cultures. So, Elwazer has built KinTrans to accommodate different types of sign language, equipping the device with language packs that make programming new signs easy. In the process, KinTrans will also create a database of regional dialects.

“We have built a system that can collect the signs and we are building the first digital library for sign language all over the world,” says Elwazer.

The device is not yet portable, but Elwazer is working with Intel’s RealSense camera technology to change that. The RealSense “depth cameras” capture the world in 3-D, like human eyes. This technology could eventually allow the hearing impaired to make phone calls without having to type out messages, as they currently must do.

“Instead of sending texts, they can call anyone. They’ll put the phone in front of them, do the signs in front of the camera, and the party on the other side of the call will hear voice,” says Elwazer.