Deep Forest-Based Monocular Visual Sign Language Recognition
Source abstract
Abstract
Sign language recognition (SLR) is a bridge linking the hearing impaired and the general public. Some SLR methods using wearable data gloves are not portable enough to provide daily sign language translation service, while visual SLR is more flexible to work with in most scenes. This paper introduces a monocular vision-based approach to SLR. Human skeleton action recognition is proposed to express semantic information, including the representation of signs’ gestures, using the regularization of body joint features and a deep-forest-based semantic classifier with a voting strategy. We test our approach on the public American Sign Language Lexicon Video Dataset (ASLLVD) and a private testing set. It proves to achieve a promising performance and shows a high generalization capability on the testing set.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Hearing Impairment and Communication · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
Funding
Natural Science Foundation of Jiangsu Province · Fundamental Research Funds for the Central Universities
References (38)
Show source-provided references
1. Sign Language Recognition Application Systems for Deaf-Mute People: A Review Based on Input-Process-Output10.1016/j.procs.2017.10.028 ↗
2. A systematic literature review on vision based gesture recognition techniques10.1007/s11042-018-5971-z ↗
3. A multimodal framework for sensor based sign language recognition10.1016/j.neucom.2016.08.132 ↗
4. Ahmed, M.A., Zaidan, B.B., Zaidan, A.A., Salih, M.M., and Bin Lakulu, M.M. (2018). A Review on Systems-Based Sensory Gloves for Sign Language Recognition State of the Art between 2007 and 2017. Sensors, 18.10.3390/s18072208 ↗
5. Wei, S., Chen, X., Yang, X., Cao, S., and Zhang, X. (2016). A Component-Based Vocabulary-Extensible Sign Language Gesture Recognition Framework. Sensors, 16.10.3390/s16040556 ↗
6. Chinese Sign Language Recognition Based on an Optimized Tree-Structure Framework10.1109/JBHI.2016.2560907 ↗
7. Su, R., Chen, X., Cao, S., and Zhang, X. (2016). Random Forest-Based Recognition of Isolated Sign Language Subwords Using Data from Accelerometers and Surface Electromyographic Sensors. Sensors, 16.10.3390/s16010100 ↗
8. Hand Gesture Recognition for Thai Sign Language in Complex Background Using Fusion of Depth and Color Video10.1016/j.procs.2016.05.113 ↗
9. Sign Language Recognition with the Kinect Sensor Based on Conditional Random Fields10.3390/s150100135 ↗
10. A Novel Phonology- and Radical-Coded Chinese Sign Language Recognition Framework Using Accelerometer and Surface Electromyography10.3390/s150923303 ↗
11. An algorithm on sign words extraction and recognition of continuous Persian sign language based on motion and shape features of hands10.1007/s10044-016-0579-2 ↗
12. A Novel Chinese Sign Language Recognition Method Based on Keyframe-Centered Clips10.1109/LSP.2018.2797228 ↗
13. Extricating Manual and Non-Manual Features for Subunit Level Medical Sign Modelling in Automatic Sign Language Classification and Recognition10.1007/s10916-017-0819-z ↗
14. Independent Bayesian classifier combination based sign language recognition using facial expression10.1016/j.ins.2017.10.046 ↗
15. Continuous sign language recognition using level building based on fast hidden Markov model10.1016/j.patrec.2016.03.030 ↗
16. Training CNNs for 3-D Sign Language Recognition With Color Texture Coded Joint Angular Displacement Maps10.1109/LSP.2018.2817179 ↗
17. Recognition of a real-time signer-independent static Farsi sign language based on fourier coefficients amplitude10.1007/s13042-016-0602-3 ↗
18. (2018, November 21). OpenPose: Real-Time Multi-Person Keypoint Detection Library for Body, Face, Hands, and Foot Estimation. Available online: https://github.com/CMU-Perceptual-Computing-Lab/openpose.
19. Zhou, Z., and Feng, J. (2017, January 19–25). Deep forest: Towards an alternative to deep neural networks. Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia.10.24963/ijcai.2017/497 ↗
20. Reshna, S., and Jayaraju, M. (2017, January 22–24). Spotting and recognition of hand gesture for Indian sign language recognition system with skin segmentation and SVM. Proceedings of the International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), Chennai, India.10.1109/WiSPNET.2017.8299784 ↗
21. An Automatic Arabic Sign Language Recognition System (ArSLRS)
22. Sparse Observation (SO) Alignment for Sign Language Recognition10.1016/j.neucom.2015.10.112 ↗
23. Dong, C., Leu, M., and Yin, Z. (2015, January 7–12). American sign language alphabet recognition using microsoft kinect. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA.
24. Feature extraction in Brazilian Sign Language Recognition based on phonological structure and using RGB-D sensors10.1016/j.eswa.2014.05.024 ↗
25. Multi-objective optimization for hand posture recognition
26. Block-based histogram of optical flow for isolated sign language recognition10.1016/j.jvcir.2016.07.020 ↗
27. Özbay, S., and Safar, M. (2017, January 21–23). Real-time sign languages recognition based on hausdorff distance, Hu invariants and neural network. Proceedings of the 2017 International Conference on Engineering and Technology (ICET), Antalya, Turkey.10.1109/ICEngTechnol.2017.8308204 ↗
28. Indian sign language recognition using graph matching on 3D motion captured signs10.1007/s11042-018-6199-7 ↗
29. Motionlets Matching with Adaptive Kernels for 3-D Indian Sign Language Recognition10.1109/JSEN.2018.2810449 ↗
30. Structured dynamic time warping for continuous hand trajectory gesture recognition10.1016/j.patcog.2018.02.011 ↗
31. A position and rotation invariant framework for sign language recognition (SLR) using Kinect10.1007/s11042-017-4776-9 ↗
32. Naresh, K. (2017, January 5–6). Sign language recognition for hearing impaired people based on hands symbols classification. Proceedings of the 2017 International Conference on Computing, Communication and Automation (ICCCA), Greater Noida, India.
33. Human-like sign-language learning method using deep learning10.4218/etrij.2018-0066 ↗
34. Spectrum of variable-random trees10.1613/jair.2470 ↗
35. From feedforward to recurrent LSTM neural networks for language modeling10.1109/TASLP.2015.2400218 ↗
36. LSTM with sentence representations for document-level sentiment classification10.1016/j.neucom.2018.04.045 ↗
37. Parsimonious memory unit for recurrent neural networks with application to natural language processing10.1016/j.neucom.2018.05.081 ↗
38. Lattice-to-sequence attentional Neural Machine Translation models10.1016/j.neucom.2018.01.010 ↗