3D Avatar Approach for Continuous Sign Movement Using Speech/Text
Source abstract
Abstract
Sign language is a visual language for communication used by hearing-impaired people with the help of hand and finger movements. Indian Sign Language (ISL) is a well-developed and standard way of communication for hearing-impaired people living in India. However, other people who use spoken language always face difficulty while communicating with a hearing-impaired person due to lack of sign language knowledge. In this study, we have developed a 3D avatar-based sign language learning system that converts the input speech/text into corresponding sign movements for ISL. The system consists of three modules. Initially, the input speech is converted into an English sentence. Then, that English sentence is converted into the corresponding ISL sentence using the Natural Language Processing (NLP) technique. Finally, the motion of the 3D avatar is defined based on the ISL sentence. The translation module achieves a 10.50 SER (Sign Error Rate) score.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Human Pose and Action Recognition · Hearing Impairment and Communication
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, Uttarakhand 247667, India
Department of Computer Science and Intelligent Systems, Osaka Prefecture University, Osaka 599-8531, Japan
Department of IT Engineering, Sookmyung Women’s University, Seoul 140-742, Korea
References (38)
Show source-provided references
1. Independent bayesian classifier combination based sign language recognition using facial expression10.1016/j.ins.2017.10.046 ↗
2. A modified LSTM model for continuous sign language recognition using leap motion10.1109/JSEN.2019.2909837 ↗
3. A multimodal framework for sensor based sign language recognition10.1016/j.neucom.2016.08.132 ↗
4. Kumar, P. (2018). Sign Language Recognition Using Depth Sensors. [Ph.D. Thesis, Indian Institute of Technology Roorkee].
5. Kumar, P., Saini, R., Behera, S.K., Dogra, D.P., and Roy, P.P. (2017, January 8–12). Real-time recognition of sign language gestures and air-writing using leap motion. Proceedings of the IEEE 2017 Fifteenth IAPR International Conference on Machine Vision Spplications (MVA), Nagoya, Japan.10.23919/MVA.2017.7986825 ↗
6. A Glove based approach to recognize Indian Sign Languages
7. Implementation of Indian Sign Language in educational settings
8. Dasgupta, T., and Basu, A. (2008, January 13–16). Prototype machine translation system from text-to-Indian sign language. Proceedings of the ACM 13th International Conference on Intelligent User Interfaces, Gran Canaria, Spain.10.1145/1378773.1378818 ↗
9. A video based Indian sign language recognition system (INSLR) using wavelet transform and fuzzy logic10.7763/IJET.2012.V4.427 ↗
10. Goyal, L., and Goyal, V. (2016, January 17–20). Automatic translation of English text to Indian sign language synthetic animations. Proceedings of the 13th International Conference on Natural Language Processing, Varanasi, India.
11. Arabic Text-to-Sign Model from Automatic SR System10.1016/j.procs.2017.10.122 ↗
12. Speech to sign language translation system for Spanish10.1016/j.specom.2008.02.001 ↗
13. Increasing adaptability of a speech into sign language translation system10.1016/j.eswa.2012.08.059 ↗
14. An avatar based translation system from arabic speech to arabic sign language for deaf people
15. Papadogiorgaki, M., Grammalidis, N., Tzovaras, D., and Strintzis, M.G. (2005, January 4–8). Text-to-sign language synthesis tool. Proceedings of the IEEE 13th European Signal Processing Conference, Antalya, Turkey.
16. Linguistic modelling and language-processing technologies for Avatar-based sign language presentation10.1007/s10209-007-0102-z ↗
17. An avatar based approach for automatic interpretation of text to Sign language
18. Loke, P., Paranjpe, J., Bhabal, S., and Kanere, K. (2017, January 20–22). Indian sign language converter system using an android app. Proceedings of the IEEE 2017 International conference of Electronics, Communication and Aerospace Technology (ICECA), Coimbatore, India.10.1109/ICECA.2017.8212852 ↗
19. Nair, M.S., Nimitha, A., and Idicula, S.M. (2016, January 1–3). Conversion of Malayalam text to Indian sign language using synthetic animation. Proceedings of the IEEE 2016 International Conference on Next Generation Intelligent Systems (ICNGIS), Kottayam, India.10.1109/ICNGIS.2016.7854002 ↗
20. Vij, P., and Kumar, P. (2016, January 12–13). Mapping Hindi Text To Indian sign language with Extension Using Wordnet. Proceedings of the International Conference on Advances in Information Communication Technology & Computing, Bikaner, India.10.1145/2979779.2979817 ↗
21. Adamo-Villani, N., Doublestein, J., and Martin, Z. (2004, January 16). The MathSigner: An interactive learning tool for American sign language. Proceedings of the IEEE Eighth International Conference on Information Visualisation, London, UK.
22. Sign transition modeling and a scalable solution to continuous sign language recognition for real-world applications10.1145/2850421 ↗
23. Translating bus information into sign language for deaf people10.1016/j.engappai.2014.02.006 ↗
24. Bouzid, Y., and Jemni, M. (2013, January 15–18). An Avatar based approach for automatically interpreting a sign language notation. Proceedings of the IEEE 13th International Conference on Advanced Learning Technologies, Beijing, China.10.1109/ICALT.2013.31 ↗
25. Duarte, A.C. (2019, January 21–25). Cross-modal neural sign language translation. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3352587 ↗
26. Patel, B.D., Patel, H.B., Khanvilkar, M.A., Patel, N.R., and Akilan, T. (September, January 30). ES2ISL: An advancement in speech to sign language translation using 3D avatar animator. Proceedings of the 2020 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), London, ON, Canada.10.1109/CCECE47787.2020.9255783 ↗
27. Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial Networks10.1007/s11263-019-01281-2 ↗
28. HamNoSys to SiGML conversion system for sign language automation10.1016/j.procs.2016.06.063 ↗
29. HamNoSys-representing sign language data in language resources and language processing contexts
30. Kennaway, R. (2015). Avatar-independent scripting for real-time gesture animation. arXiv.
31. Zhang, Y., Vogel, S., and Waibel, A. (2019, January 26–28). Interpreting bleu/nist scores: How much improvement do we need to have a better system?. Proceedings of the Fourth International Conference on Language Resources and Evaluation, Lisbon, Portugal.
32. Spoken content retrieval: A survey of techniques and technologies10.1561/1500000020 ↗
33. Spoken content retrieval—Beyond cascading speech recognition with text retrieval10.1109/TASLP.2015.2438543 ↗
34. Bird, S., Klein, E., and Loper, E. (2009). Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit, O’Reilly Media Inc.
35. Automated 3D sign language caption generation for video10.1007/s10209-019-00668-9 ↗
36. Indo-Pakistani Sign Language grammar: A typological outline10.1353/sls.2003.0005 ↗
37. Roosendaal, T., and Wartmann, C. (2003). The Official Blender Game Kit: Interactive 3d for Artists, No Starch Press.
38. Tominaga, T., Hayashi, T., Okamoto, J., and Takahashi, A. (2010, January 21–23). Performance comparisons of subjective quality assessment methods for mobile video. Proceedings of the IEEE 2010 Second international workshop on quality of multimedia experience (QoMEX), Trondheim, Norway.10.1109/QOMEX.2010.5517948 ↗