Neural Sign Language Translation Based on Human Keypoint Estimation
Source abstract
Abstract
We propose a sign language translation system based on human keypoint estimation. It is well-known that many problems in the field of computer vision require a massive dataset to train deep neural network models. The situation is even worse when it comes to the sign language translation problem as it is far more difficult to collect high-quality training data. In this paper, we introduce the KETI (Korea Electronics Technology Institute) sign language dataset, which consists of 14,672 videos of high resolution and quality. Considering the fact that each country has a different and unique sign language, the KETI sign language dataset can be the starting point for further research on the Korean sign language translation. Using the KETI sign language dataset, we develop a neural network model for translating sign videos into natural language sentences by utilizing the human keypoints extracted from the face, hands, and body parts. The obtained human keypoint vector is normalized by the mean and standard deviation of the keypoints and used as input to our translation model based on the sequence-to-sequence architecture. As a result, we show that our approach is robust even when the size of the training data is not sufficient. Our translation model achieved 93.28% (55.28%, respectively) translation accuracy on the validation set (test set, respectively) for 105 sentences that can be used in emergency situations. We compared several types of our neural sign translation models based on different attention mechanisms in terms of classical metrics for measuring the translation performance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Human Pose and Action Recognition · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Korea Electronics Technology Institute, Seongnam 13488, Korea
Funding
Institute for Information and communications Technology Promotion
References (59)
Show source-provided references
1. Forster, J., Schmidt, C., Hoyoux, T., Koller, O., Zelle, U., Piater, J., and Ney, H. (2012, January 23–25). RWTH-PHOENIX-Weather: A Large Vocabulary Sign Language Recognition and Translation Corpus. Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC 2012), Istanbul, Turkey.
2. Von Agris, U., Knorr, M., and Kraiss, K.F. (2008, January 17–19). The Significance of Facial Features for Automatic Sign Language Recognition. Proceedings of the 8th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2008), Amsterdam, The Netherlands.10.1109/AFGR.2008.4813472 ↗
3. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
4. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014, January 25–29). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), Doha, Qatar.10.3115/v1/D14-1179 ↗
5. Dai, B., and Lin, D. (2017, January 4–9). Contrastive Learning for Image Captioning. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 (NIPS 2017), Long Beach, CA, USA.
6. Liu, C., Mao, J., Sha, F., and Yuille, A.L. (2017, January 4–9). Attention Correctness in Neural Image Captioning. Proceedings of the 31th AAAI Conference on Artificial Intelligence (AAAI 2017), San Francisco, CA, USA.10.1609/aaai.v31i1.11197 ↗
7. Sutskever, I., Vinyals, O., and Le, Q.V. (2014, January 8–13). Sequence to Sequence Learning with Neural Networks. Proceedings of the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014 (NIPS 2014), Montréal, Canada.
8. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015, January 6–11). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), Lille, France.
9. Hu, J., Shen, L., and Sun, G. (2018, January 18–22). Squeeze-and-Excitation Networks. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
10. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
11. Gao, M., Yu, R., Li, A., Morariu, V.I., and Davis, L.S. (2017, January 21–26). Dynamic zoom-in network for fast object detection in large images. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.
12. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
13. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015), Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
14. Zhang, H., Dana, K., Shi, J., Zhang, Z., Wang, X., Tyagi, A., and Agrawal, A. (2018, January 18–22). Context encoding for semantic segmentation. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA.10.1109/CVPR.2018.00747 ↗
15. Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. (2015, January 7–12). Long-term Recurrent Convolutional Networks for Visual Recognition and Description. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015), Boston, MA, USA.10.1109/CVPR.2015.7298878 ↗
16. Luvizon, D.C., Picard, D., and Tabia, H. (2018, January 18–22). 2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA.10.1109/CVPR.2018.00539 ↗
17. Dong, C., Leu, M.C., and Yin, Z. (2015, January 7–12). American Sign Language alphabet recognition using Microsoft Kinect. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015), Boston, MA, USA.
18. Gattupalli, S., Ghaderi, A., and Athitsos, V. (July, January 29). Evaluation of deep learning based pose estimation for sign language recognition. Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA 2016), Corfu, Greece.10.1145/2910674.2910716 ↗
19. Kishore, P.V.V., Sastry, A.S.C.S., and Kartheek, A. (2014, January 7–9). Visual-verbal machine interpreter for sign language recognition under versatile video backgrounds. Proceedings of the 1st International Conference on Networks Soft Computing (ICNSC 2014), Miami, FL, USA.10.1109/CNSC.2014.6906696 ↗
20. Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers10.1016/j.cviu.2015.09.013 ↗
21. Koller, O., Zargaran, S., and Ney, H. (2017, January 21–26). Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent CNN-HMMs. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.10.1109/CVPR.2017.364 ↗
22. Starner, T., and Pentland, A. (1995, January 21–23). Real-time American Sign Language recognition from video using hidden Markov models. Proceedings of the International Symposium on Computer Vision (ISCV 1995), Coral Gables, FL, USA.
23. Cihan Camgoz, N., Hadfield, S., Koller, O., Ney, H., and Bowden, R. (2018, January 18–22). Neural Sign Language Translation. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA.10.1109/CVPR.2018.00812 ↗
24. Kim, T., and Kim, S. (2016, January 19–22). Sign language translation system using latent feature values of sign language images. Proceedings of the 13th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI 2016), Xi’an, China.10.1109/URAI.2016.7625743 ↗
25. Buehler, P., Zisserman, A., and Everingham, M. (2009, January 20–25). Learning sign language by watching TV (using weakly aligned subtitles). Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2009), Miami Beach, FL, USA.10.1109/CVPRW.2009.5206523 ↗
26. Cooper, H., and Bowden, R. (2009, January 20–25). Learning signs from subtitles: A weakly supervised approach to sign language recognition. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2009), Miami Beach, FL, USA.10.1109/CVPRW.2009.5206647 ↗
27. Pfister, T., Charles, J., and Zisserman, A. (2013, January 9–13). Large-scale Learning of Sign Language by Watching TV (Using Co-occurrences). Proceedings of the 24th British Machine Vision Conference (BMVC 2013), Bristol, UK.10.5244/C.27.20 ↗
28. Forster, J., Schmidt, C., Koller, O., Bellgardt, M., and Ney, H. (2014, January 26–31). Extensions of the Sign Language Recognition and Translation Corpus RWTH-PHOENIX-Weather. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014), Reykjavik, Iceland.
29. Oberweger, M., Wohlhart, P., and Lepetit, V. (2015, January 9–11). Hands Deep in Deep Learning for Hand Pose Estimation. Proceedings of the 20th Computer Vision Winter Workshop (CVWW 2015), Styria, Austria.
30. Ko, S., Son, J.G., and Jung, H.D. (2018, January 9–12). Sign language recognition with recurrent neural network using human keypoint detection. Proceedings of the 2018 Conference on Research in Adaptive and Convergent Systems (RACS 2018), Honolulu, HI, USA.10.1145/3264746.3264805 ↗
31. Liddell, S.K. (2003). Grammar, Gesture, and Meaning in American Sign Language, Cambridge University Press.10.1017/CBO9780511615054 ↗
32. Cao, Z., Simon, T., Wei, S., and Sheikh, Y. (2017, January 21–26). Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
33. Simon, T., Joo, H., Matthews, I.A., and Sheikh, Y. (2017, January 21–26). Hand Keypoint Detection in Single Images Using Multiview Bootstrapping. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.10.1109/CVPR.2017.494 ↗
34. Wei, S., Ramakrishna, V., Kanade, T., and Sheikh, Y. (2016, January 27–30). Convolutional Pose Machines. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), Las Vegas, NV, USA.10.1109/CVPR.2016.511 ↗
35. Ba, L.J., Kiros, R., and Hinton, G.E. (2016). Layer Normalization. arXiv.
36. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), Lille, France.
37. Ulyanov, D., Vedaldi, A., and Lempitsky, V.S. (2016). Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv.
38. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014, January 23–28). Large-Scale Video Classification with Convolutional Neural Networks. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014), Columbus, OH, USA.10.1109/CVPR.2014.223 ↗
39. Perez, L., and Wang, J. (2017). The Effectiveness of Data Augmentation in Image Classification using Deep Learning. arXiv.
40. Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7–9). Neural Machine Translation by Jointly Learning to Align and Translate. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA.
41. Luong, T., Pham, H., and Manning, C.D. (2015, January 17–21). Effective Approaches to Attention-based Neural Machine Translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015), Lisbon, Portugal.10.18653/v1/D15-1166 ↗
42. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 (NIPS 2017), Long Beach, CA, USA.
43. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2019, June 30). Automatic Differentiation in PyTorch. Available online: https://openreview.net/pdf?id=BJJsrmfCZ.
44. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA.
45. Park, E.L., and Cho, S. (2014, January 11–14). KoNLPy: Korean natural language processing in Python. Proceedings of the 26th Annual Conference on Human & Cognitive Language Technology (HCLT 2014), Chuncheon, Korea.
46. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6–12). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL 2002), Philadephia, PA, USA.10.3115/1073083.1073135 ↗
47. Lin, C.Y. (2004, January 25–26). ROUGE: A Package for Automatic Evaluation of summaries. Proceedings of the ACL workshop on Text Summarization Branches Out, Barcelona, Spain.
48. Banerjee, S., and Lavie, A. (2005, January 29). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA.
49. Vedantam, R., Zitnick, C.L., and Parikh, D. (2015, January 7–12). CIDEr: Consensus-based image description evaluation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015), Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
50. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
51. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA.
52. Pascanu, R., Mikolov, T., and Bengio, Y. (2013, January 16–21). On the difficulty of training recurrent neural networks. Proceedings of the 30th International Conference on Machine Learning (ICML 2013), Atlanta, GA, USA.
53. Hoffer, E., Hubara, I., and Soudry, D. (2017, January 4–9). Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
54. Smith, S.L., Kindermans, P., Ying, C., and Le, Q.V. (May, January 30). Don’t Decay the Learning Rate, Increase the Batch Size. Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), Vancouver, BC, Canada.
55. He, K., Gkioxari, G., Dollár, P., and Girshick, R.B. (2017, January 22–29). Mask R-CNN. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV 2017), Venice, Italy.10.1109/ICCV.2017.322 ↗
56. Fang, H., Xie, S., Tai, Y., and Lu, C. (2017, January 22–29). RMPE: Regional Multi-person Pose Estimation. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV 2017), Venice, Italy.10.1109/ICCV.2017.256 ↗
57. Duan, J., Schlemper, J., Bai, W., Dawes, T.J.W., Bello, G., Doumou, G., De Marvao, A., O’Regan, D.P., and Rueckert, D. (2018, January 16–20). Deep Nested Level Sets: Fully Automated Segmentation of Cardiac MR Images in Patients with Pulmonary Hypertension. Proceedings of the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2018), Granada, Spain.10.1007/978-3-030-00937-3_68 ↗
58. Duan, J., Bello, G., Schlemper, J., Bai, W., Dawes, T., Biffi, C., de Marvao, A., Doumou, G., O’Regan, D., and Rueckert, D. (2019). Automatic 3D bi-ventricular segmentation of cardiac images by a shape-refined multi-task deep learning approach. IEEE Trans. Med. Imag.10.1109/TMI.2019.2894322 ↗
59. Facial Landmark Detection: A Literature Survey10.1007/s11263-018-1097-z ↗