Kinect used in sign-language recognition and translation

In a collaboration with Microsoft Research Asia and the Institute of Computing Technology at the Chinese Academy of Sciences (CAS) researchers tested how the Kinect's body-tracking features can be used in sign-language recognition, according to the official Inside Microsoft Research website.

According to the blog, initial results show that the technology built from Kinect for Windows can help those who use sign language as primary language to better interact with computers, similar to speech recognition software.

The system features a Translation Mode that translates sign language into text or speech, which currently supports American sign language with the capacity to support other sign language varieties.

Using an avatar, the system's Communications Mode allows a deaf or hard-of-hearing person to communicate with a hearing person. The avatar acts out the text input from a keyboard and when the deaf or hard-of-hearing person responds with sign language, it is converted into text.

"From our point of view," says CAS Professor Xilin Chen told the blog. "The most significant contribution is that the project demonstrates the possibility of sign-language recognition with readily available, low-cost 3-D and 2-D sensors."

The research features in the paper "Sign Language Recognition and Translation with Kinect." The paper was co-authored by CAS researchers Xiujuan Chai, Guang Li, Yushun Lin, Zhihao Xu, Yili Tang, and Chen, as well as principal researcher at Microsoft Research Asia, Ming Zhou. The project was supported by Microsoft Research Connections.

You can watch the technology in action in the video above.

More from Polygon

The horror of Five Nights at Freddy's

  • Dark Souls 2: Crown of the Old Iron King Overview

  • Spacecom: a fast 4X built for multiplayer

  • Pillars of Eternity builds on role-playing classics

  • Tour the 1 KB hard drive built inside Minecraft

Latest Discussions

Log In Sign Up

Log In Sign Up

Please choose a new Polygon username and password

As part of the new Polygon launch, prior users will need to choose a permanent username, along with a new password.

Your username will be used to login to Polygon going forward.

I already have a Vox Media account!

Verify Vox Media account

Please login to your Vox Media account. This account will be linked to your previously existing Eater account.

Please choose a new Polygon username and password

As part of the new Polygon launch, prior MT authors will need to choose a new username and password.

Your username will be used to login to Polygon going forward.

Forgot password?

We'll email you a reset link.

If you signed up using a 3rd party account like Facebook or Twitter, please login with it instead.

Forgot password?

Try another email?

Almost done,

By becoming a registered user, you are also agreeing to our Terms and confirming that you have read our Privacy Policy.



Choose an available username to complete sign up.

In order to provide our users with a better overall experience, we ask for more information from Facebook when using it to login so that we can learn more about our audience and provide you with the best possible experience. We do not store specific user data and the sharing of it is not required to login with Facebook.