Deaf programmer from Novosibirsk developed an automatic translator from sign language

Deaf programmer Alexei Prikhodko developed a system that automatically translates sign language into Russian. The technology allows you to control a computer without using a mouse. He finished work on the prototype in early August.       The program camera recognizes the picture and captures gestures, then the image is transferred to the model and processed inside the system. The system compares the picture with the data in the neural network, after which the computer displays the translation that already corresponds to the gestures on the monitor, according to the website of Novosibirsk State Technical University.    According to Alexei Prikhodko, at the moment there are many companies that declare their own development of a translator for the deaf, and some even sell their solutions. But, due to the specifics of sign language, all of them are not high enough for use, says a programmer who himself has not heard since childhood.        “Savvy Motion, the Kinect Sign Language Translator at Microsoft Research and other large companies have not yet been able to fully realize this task of translating from sign language to sound language, so the quality of such applications leaves much to be desired,” he says. The problem is related to the grammar of sign language. No program can replace a live translator, because the translation directly depends not only on the configuration and orientation of the hands, but also on their movement, location and the so-called linguists non-manual component of gestures (facial expression, lip movement and other signs of articulation), reported on University website.    “It is not difficult to translate from written language to sign language. It is technically difficult to recognize gestures and translate them. It all depends on which camera and which sensors. There are two ways to recognize gestures using technology: markerless and marker. The marker system is when a person wears special gloves, devices on his wrists, bracelets and modern units that take into account the movement of muscles and points on the body of a person.   I went the difficult way, which does not require large expenditures on special equipment to create a program with a marker system. My program recognizes a person and his gestures using cameras using a markerless method, ”said Alexei Prikhodko.     The markerless system created by the programmer, using special cameras, imposes a virtual “grid” on the image. On it, software algorithms find reference points by which gestures are determined. Then the system processes the data and reproduces the specified actions: translation or control.    “If the model determines, for example, that the fingers are open – the letter B, if the fingers are assembled – O. The elbow is bent or not. Depending on this, a certain mathematical model is formed, which is created from a skeletal model. And accordingly, each number from this model is assigned a coordinate system, and on the screen we see what kind of gesture it is, ”the programmer said. Currently, the prototype translates at the ABC level of the deaf. To defend his dissertation, Alexei Prikhodko intends to teach the system other components of sign language grammar in order to create a ready-made product for mass introduction and use by people with hearing impairment. To refine the prototype, the programmer is looking for investors.       Alexei Prikhodko is the only programmer and expert in the world to develop a system for translating sign language, according to the website of Novosibirsk State Technical University. Previously, he already created technical solutions for people with hearing impairment. In early July 2019, he and his team created a project for teaching the deaf – “Gesture Interface”, in 2017 – the project “Mathematics in Silence”.    According to Inc., scientists from the University of California at San Diego previously developed a smart glove that automatically translates American Sign Language (ASL) into text on a digital device’s screen. The cost of the gadget did not exceed $ 100.