Recognition of American Sign Language Gestures in a Virtual Reality Using Leap Motion
Source abstract
Abstract
We perform gesture recognition in a Virtual Reality (VR) environment using data produced by the Leap Motion device. Leap Motion generates a virtual three-dimensional (3D) hand model by recognizing and tracking user‘s hands. From this model, the Leap Motion application programming interface (API) provides hand and finger locations in the 3D space. We present a system that is capable of learning gestures by using the data from the Leap Motion device and the Hidden Markov classification (HMC) algorithm. We have achieved the gesture recognition accuracy (mean ± SD) is 86.1 ± 8.2% and gesture typing speed is 3.09 ± 0.53 words per minute (WPM), when recognizing the gestures of the American Sign Language (ASL).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Hearing Impairment and Communication · Tactile and Sensory Interactions
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Software Engineering, Kaunas University of Technology, 50186 Kaunas, Lithuania
Department of Multimedia Engineering, Kaunas University of Technology, 50186 Kaunas, Lithuania
Institute of Mathematics, Silesian University of Technology, 44-100 Gliwice, Poland
References (50)
Show source-provided references
1. Bachmann, D., Weichert, F., and Rinkenauer, G. (2018). Review of three-dimensional human-computer interaction with focus on the leap motion controller. Sensors, 18.10.3390/s18072194 ↗
2. Dawes, F., Penders, J., and Carbone, G. (2019). Remote Control of a Robotic Hand Using a Leap Sensor. The International Conference of IFToMM ITALY, Springer International Publishing.10.1007/978-3-030-03320-0_36 ↗
3. Playing into the wild: A gesture-based interface for gaming in public spaces10.1016/j.jvcir.2011.12.006 ↗
4. VRCDEA-TCS: 3D virtual reality cooperative drawing educational application with textual chatting system10.1002/cae.22017 ↗
5. Sign Language Recognition with the Kinect Sensor Based on Conditional Random Fields10.3390/s150100135 ↗
6. Objective and automatic classification of parkinson disease with leap motion controller10.1186/s12938-018-0600-7 ↗
7. A case study of gesture-based games in enhancing the fine motor skills and recognition of children with autism10.1080/10494820.2018.1437048 ↗
8. Cohen, M.W., Voldman, I., Regazzoni, D., and Vitali, A. (2018, January 4–6). Hand rehabilitation via gesture recognition using leap motion controller. Proceedings of the 11th International Conference on Human System Interaction, HIS, Gdansk, Poland.10.1109/HSI.2018.8431349 ↗
9. Morando, M., Ponte, S., Ferrara, E., and Dellepiane, S. (2018). Definition of motion and biophysical indicators for home-based rehabilitation through serious games. Information, 9.10.3390/info9050105 ↗
10. Qingchao, X., and Jiangang, C. (2017). The application of leap motion in astronaut virtual training. Iop Conf. Ser. Mater. Sci. Eng., 187.10.1088/1757-899X/187/1/012015 ↗
11. VR surgery: Interactive virtual reality application for training oral and maxillofacial surgeons using oculus rift and leap motion10.1007/978-3-319-51645-5_8 ↗
12. Gleeson, B., MacLean, K., Haddadi, A., Croft, E., and Alcazar, J. (2013, January 3–6). Gestures for industry: Intuitive human-robot communication from human observation. Proceedings of the 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI ’13), Tokyo, Japan.10.1109/HRI.2013.6483609 ↗
13. Mohandes, M., Aliyu, S., and Deriche, M. (2014). Arabic sign language recognition using the leap motion controller. IEEE Int. Symp. Ind. Electron., 960–965.10.1109/ISIE.2014.6864742 ↗
14. Alfonse, M., Ali, A., Elons, A.S., Badr, N.L., and Aboul-Ela, M. (2015, January 21–23). Arabic sign language benchmark database for different heterogeneous sensors. Proceedings of the 2015 5th International Conference on Information & Communication Technology and Accessibility (ICTA), Marrakech, Morocco.10.1109/ICTA.2015.7426902 ↗
15. Chavan, P., Ghorpade, T., and Padiya, P. (2016, January 18–19). Indian sign language to forecast text using leap motion sensor and RF classifier. Proceedings of the 2016 Symposium on Colossal Data Analysis and Networking (CDAN), Indore, India.10.1109/CDAN.2016.7570936 ↗
16. Naglot, D., and Kulkarni, M. (2016, January 26–27). ANN based indian sign language numerals recognition using the leap motion controller. Proceedings of the International Conference on Inventive Computation Technologies, ICICT 2016, Coimbatore, India.10.1109/INVENTIVE.2016.7824830 ↗
17. Demircioǧlu, B., Bülbül, G., and Köse, H. (2016, January 16–19). Turkish sign language recognition with leap motion. Proceedings of the 2016 24th Signal Processing and Communication Application Conference (SIU), Zonguldak, Turkey.10.1109/SIU.2016.7495809 ↗
18. Simos, M., and Nikolaidis, N. (2016, January 18–20). Greek sign language alphabet recognition using the leap motion device. Proceedings of the 9th Hellenic Conference on Artificial Intelligence, Thessaloniki, Greece.10.1145/2903220.2903249 ↗
19. Tumsri, J., and Kimpan, W. (2017, January 15–17). Thai sign language translation using leap motion controller. Proceedings of the International Multi Conference of Engineers and Computer Scientists, Hong Kong, China.
20. Anwar, A., Basuki, A., Sigit, R., Rahagiyanto, A., and Zikky, M. (2017, January 15–18). Feature extraction for indonesian sign language (SIBI) using leap motion controller. Proceedings of the 2017 21st International Computer Science and Engineering Conference (ICSEC), Bangkok, Thailand.10.1109/ICSEC.2017.8443926 ↗
21. Chuan, C., Regina, E., and Guardino, C. (2014, January 3–6). American sign language recognition using leap motion sensor. Proceedings of the 2014 13th International Conference on Machine Learning and Applications, Detroit, MI, USA.10.1109/ICMLA.2014.110 ↗
22. Mapari, R.B., and Kharat, G. (2016, January 4–5). American static signs recognition using leap motion sensor. Proceedings of the Second International Conference on Information and Communication Technology for Competitive Strategies, Udaipur, India.10.1145/2905055.2905125 ↗
23. Chong, T., and Lee, B. (2018). American sign language recognition using leap motion controller with machine learning approach. Sensors, 18.10.3390/s18103554 ↗
24. Avola, D., Bernardi, M., Cinque, L., Foresti, G.L., and Massaroni, C. (2018). Exploiting recurrent neural networks and leap motion controller for the recognition of sign language and semaphoric hand gestures. IEEE Trans. Multimed.10.1109/TMM.2018.2856094 ↗
25. Ameur, S., Khalifa, A.B., and Bouhlel, M.S. (2016, January 18–20). A comprehensive leap motion database for hand gesture recognition. Proceedings of the 2016 7th International Conference on Sciences of Electronics, Technologies of Information and Telecommunications (SETIT), Hammamet, Tunisia.10.1109/SETIT.2016.7939924 ↗
26. Fok, K., Ganganath, N., Cheng, C., and Tse, C.K. (2015, January 17–19). A real-time ASL recognition system using leap motion sensors. Proceedings of the 2015 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, Xi’an, China.10.1109/CyberC.2015.81 ↗
27. Arabic static and dynamic gestures recognition using leap motion10.3844/jcssp.2017.337.354 ↗
28. Dynamic hand gesture recognition with leap motion controller10.1109/LSP.2016.2590470 ↗
29. Contact dynamics emulation using leap motion controller10.1007/978-981-10-5427-3_28 ↗
30. Augmented Reality & Gesture-based Architecture in Games for the Elderly
31. Sourial, M., Elnaggar, A., and Reichardt, D. (2017, January 6–7). Development of a virtual coach scenario for hand therapy using LEAP motion. Proceedings of the 2016 Future Technologies Conference (FTC), San Francisco, CA, USA.10.1109/FTC.2016.7821736 ↗
32. Accuracy in fingertip tracking using leap motion controller for interactive virtual applications10.1007/s12008-016-0339-y ↗
33. Proposed system on gesture controlled holographic projection using leap motion
34. Jimenez, J.G., and Schulze, J.P. (2018). Continuous-Motion Text Input in Virtual Reality. Electron. Imaging, 450-1–450-6.10.2352/ISSN.2470-1173.2018.03.ERVR-450 ↗
35. Komiya, K., and Nakajima, T. (2017, January 3–5). A Japanese input method using leap motion in virtual reality. Proceedings of the Tenth International Conference on Mobile Computing and Ubiquitous Network (ICMU), Toyama, Japan.10.23919/ICMU.2017.8330085 ↗
36. Virtual grasps recognition using fusion of Leap Motion and force myography10.1007/s10055-018-0339-2 ↗
37. Lee, Y.S., and Bong-Soo Sohn, B.S. (2018). Immersive Gesture Interfaces for Navigation of 3D Maps in HMD-Based Mobile Virtual Environments. Mob. Inf. Syst., 2585797.10.1109/ICOIN.2018.8343267 ↗
38. Hand gesture recognition using hidden markov models
39. Microservices in Practice, Part 1: Reality Check and Service Design10.1109/MS.2017.24 ↗
40. Preventis, A., Stravoskoufos, K., Sotiriadis, S., and Petrakis, E.G.M. (2014, January 8–11). Interact: Gesture Recognition in the Cloud. Proceedings of the IEEE/ACM 7th International Conference on Utility and Cloud Computing, London, UK.10.1109/UCC.2014.71 ↗
41. Hand gesture recognition with jointly calibrated leap motion and depth sensor10.1007/s11042-015-2451-6 ↗
42. Sub-millimetre accurate human hand kinematics: From surface to skeleton10.1080/10255842.2018.1425996 ↗
43. Fink, G.-A. (2008). Markov Models for Pattern Recognition, Springer. [2nd ed.].
44. A tutorial on Hidden Markov Models and selected applications in speech recognition
45. Liddell, S.K. (2003). Grammar, Gesture, and Meaning in American Sign Language, Cambridge University Press.10.1017/CBO9780511615054 ↗
46. Dobosz, K., and Buchczyk, K. (2018). One-Handed Braille in the Air. International Conference on Computers Helping People with Special Needs ICCHP 2018, Lecture Notes in Computer Science, Springer International Publishing.10.1007/978-3-319-94274-2_44 ↗
47. A Systematic Analysis of Performance Measures for Classification Tasks10.1016/j.ipm.2009.03.002 ↗
48. A Coefficient of Agreement for Nominal Scales10.1177/001316446002000104 ↗
49. Soukoreff, R.W., and MacKenzie, I.S. (2003, January 5–10). Metrics for text entry research: An evaluation of MSD and KSPC, and a new unified error metric. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Ft. Lauderdale, FL, USA.10.1145/642611.642632 ↗
50. Sign Language Recognition Systems: A Review