Korean Sign Language Recognition Using Transformer-Based Deep Neural Network
Source abstract
Abstract
Sign language recognition (SLR) is one of the crucial applications of the hand gesture recognition and computer vision research domain. There are many researchers who have been working to develop a hand gesture-based SLR application for English, Turkey, Arabic, and other sign languages. However, few studies have been conducted on Korean sign language classification because few KSL datasets are publicly available. In addition, the existing Korean sign language recognition work still faces challenges in being conducted efficiently because light illumination and background complexity are the major problems in this field. In the last decade, researchers successfully applied a vision-based transformer for recognizing sign language by extracting long-range dependency within the image. Moreover, there is a significant gap between the CNN and transformer in terms of the performance and efficiency of the model. In addition, we have not found a combination of CNN and transformer-based Korean sign language recognition models yet. To overcome the challenges, we proposed a convolution and transformer-based multi-branch network aiming to take advantage of the long-range dependencies computation of the transformer and local feature calculation of the CNN for sign language recognition. We extracted initial features with the grained model and then parallelly extracted features from the transformer and CNN. After concatenating the local and long-range dependencies features, a new classification module was applied for the classification. We evaluated the proposed model with a KSL benchmark dataset and our lab dataset, where our model achieved 89.00% accuracy for 77 label KSL dataset and 98.30% accuracy for the lab dataset. The higher performance proves that the proposed model can achieve a generalized property with considerably less computational cost.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Hearing Impairment and Communication · Gait Recognition and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, The University of Aizu, Aizuwakamatsu 965-8580, Fukushima, Japan
Department of Computer Science and Engineering, Rajshahi University of Engineering and Technology (RUET), Rajshahi 6204, Bangladesh
Department of Applied Software Engineering, Dongeui University, Busanjin-Gu, Busan 47340, Republic of Korea
Department of Computer Engineering, Dongeui University, Busanjin-Gu, Busan 47340, Republic of Korea
Funding
MSIT (Ministry of Science and ICT), Korea · The University of Aizu
References (86)
Show source-provided references
1. Rafi, A.M., Nawal, N., Bayev, N.S.N., Nima, L., Shahnaz, C., and Fattah, S.A. (2019, January 17–20). Image-based bengali sign language alphabet recognition for deaf and dumb community. Proceedings of the 2019 IEEE Global Humanitarian Technology Conference (GHTC), Seattle, WA, USA.10.1109/GHTC46095.2019.9033031 ↗
2. Dynamic Hand Gesture Recognition using Multi-Branch Attention Based Graph and General Deep Learning Model10.1109/ACCESS.2023.3235368 ↗
3. Miah, A.S.M., Hasan, M.A.M., Shin, J., Okuyama, Y., and Tomioka, Y. (2023). Multistage Spatial Attention-Based Neural Network for Hand Gesture Recognition. Computers, 12.10.3390/computers12010013 ↗
4. Rotation, Translation And Scale Invariant Sign Word Recognition Using Deep Learning10.32604/csse.2023.029336 ↗
5. Miah, A.S.M., Shin, J., Hasan, M.A.M., and Rahim, M.A. (2022). BenSignNet: Bengali Sign Language Alphabet Recognition Using Concatenated Segmentation and Convolutional Neural Network. Appl. Sci., 12.10.3390/app12083933 ↗
6. Miah, A.S.M., Rahim, M.A., and Shin, J. (2020). Motor-imagery classification using riemannian geometry with median absolute deviation. Electronics, 9.10.3390/electronics9101584 ↗
7. Miah, A.S.M., Shin, J., Islam, M.M., and Molla, M.K.I. (2022, January 10–12). Natural Human Emotion Recognition Based on Various Mixed Reality (MR) Games and Electroencephalography (EEG) Signals. Proceedings of the 2022 IEEE 5th Eurasian Conference on Educational Innovation (ECEI), Taipei, Taiwan.10.1109/ECEI53102.2022.9829482 ↗
8. Rahim, M.A., Miah, A.S.M., Sayeed, A., and Shin, J. (2020, January 22). Hand Gesture Recognition Based on Optimal Segmentation in Human-Computer Interaction. Proceedings of the 2020 3rd IEEE International Conference on Knowledge Innovation and Invention (ICKII), Hualien, Taiwan.10.1109/ICKII50300.2020.9318870 ↗
9. Miah, A.S.M., Mamunur Rashid, M., Rahman, R., Hossain, T., Sujon, S., Nawal, N., Hasan, M., and Shin, J. (2021). Proceedings of the International Conference on Intelligent Computing & Optimization, Springer.
10. Skala, V., Singh, T.P., Choudhury, T., Tomar, R., and Abul Bashar, M. (2022). Proceedings of the Machine Intelligence and Data Science Applications, Springer Nature.10.1007/978-981-19-2347-0 ↗
11. Miah, A.S.M., Mouly, M.A., Debnath, C., Shin, J., and Sadakatul Bari, S. (2021). Proceedings of the International Conference on Computing Science, Communication and Security, Springer.
12. Cui, R., Liu, H., and Zhang, C. (2017, January 21–26). Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.175 ↗
13. Koller, O., Zargaran, S., and Ney, H. (2017, January 21–26). Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent CNN-HMMs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.364 ↗
14. Camgoz, N.C., Hadfield, S., Koller, O., Ney, H., and Bowden, R. (2018, January 18–22). Neural sign language translation. Proceedings of the Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00812 ↗
15. Camgoz, N.C., Koller, O., Hadfield, S., and Bowden, R. (2020, January 14–19). Sign language transformers: Joint end-to-end sign language recognition and translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.
16. Wearable sensor-based sign language recognition: A comprehensive review10.1109/RBME.2020.3019769 ↗
17. Sharma, S., and Singh, S. (2022, January 20–22). Vision-based sign language recognition system: A Comprehensive Review. Proceedings of the 2020 International Conference on Inventive Computation Technologies (ICICT), Lalitpur, Nepal.10.1109/ICICT48043.2020.9112409 ↗
18. Rajan, R.G., and Leo, M.J. (2020, January 26–28). American sign language alphabets recognition using hand-crafted and deep learning features. Proceedings of the 2020 International Conference on Inventive Computation Technologies (ICICT), Coimbatore, India.10.1109/ICICT48043.2020.9112481 ↗
19. Podder, K.K., Chowdhury, M.E., Tahir, A.M., Mahbub, Z.B., Khandakar, A., Hossain, M.S., and Kadir, M.A. (2022). Bangla sign language (bdsl) alphabets and numerals classification using a deep learning model. Sensors, 22.10.3390/s22020574 ↗
20. Awan, M.J., Rahim, M.S.M., Salim, N., Rehman, A., Nobanee, H., and Shabir, H. (2021). Improved deep convolutional neural network to classify osteoarthritis from anterior cruciate ligament tear using magnetic resonance imaging. J. Pers. Med., 11.10.3390/jpm11111163 ↗
21. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
22. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
23. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
24. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–22). Non-local neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
25. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020). Proceedings of the European Conference on Computer Vision, Springer.
26. Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. (2019, January 8–14). Stand-alone self-attention in vision models. Proceedings of the Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Vancouver, BC, Canada.
27. Wang, H., Zhu, Y., Green, B., Adam, H., Yuille, A., and Chen, L.C. (2020). Proceedings of the European Conference on Computer Vision, Springer.
28. Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., and Xu, C. (2022, January 19–26). Cmt: Convolutional neural networks meet vision transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01186 ↗
29. De Coster, M., Van Herreweghe, M., and Dambre, J. (2020, January 11–16). Sign language recognition with transformer networks. Proceedings of the 12th International Conference on Language Resources and Evaluation, Marseille, France.
30. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
31. Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., and Gao, W. (2021, January 19–25). Pre-trained image processing transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01212 ↗
32. Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., and Torr, P.H. (2021, January 19–25). Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00681 ↗
33. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2021, January 18–24). Training data-efficient image transformers & distillation through attention. Proceedings of the International Conference on Machine Learning, Online.
34. Automatic sign language analysis: A survey and the future beyond lexical meaning10.1109/TPAMI.2005.112 ↗
35. Madhiarasan, D.M., Roy, P., and Pratim, P. (2022). A Comprehensive Review of Sign Language Recognition: Different Types, Modalities, and Datasets. arXiv.
36. Uddin, M.A., and Chowdhury, S.A. (2016, January 28–29). Hand sign language recognition for bangla alphabet using support vector machine. Proceedings of the 2016 International Conference on Innovations in Science, Engineering and Technology (ICISET), Chittagong, Bangladesh.10.1109/ICISET.2016.7856479 ↗
37. Yasir, F., Prasad, P., Alsadoon, A., Elchouemi, A., and Sreedharan, S. (2017, January 6–7). Bangla Sign Language recognition using convolutional neural network. Proceedings of the 2017 International Conference on Intelligent Computing, Instrumentation and Control Technologies (ICICICT), Kerala, India.10.1109/ICICICT1.2017.8342533 ↗
38. Abedin, T., Prottoy, K.S., Moshruba, A., and Hakim, S.B. (2021). Bangla sign language recognition using concatenated BdSL network. arXiv.
39. Pitsikalis, V., Theodorakis, S., Vogler, C., and Maragos, P. (2011, January 20–25). Advances in phonetics-based sub-unit modeling for transcription alignment and sign language recognition. Proceedings of the CVPR 2011 WORKSHOPS, Colorado Springs, CO, USA.10.1109/CVPRW.2011.5981681 ↗
40. Ong, E.J., Cooper, H., Pugeault, N., and Bowden, R. (2012, January 16–21). Sign language recognition using sequential pattern trees. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.
41. Feature extraction in Brazilian Sign Language Recognition based on phonological structure and using RGB-D sensors10.1016/j.eswa.2014.05.024 ↗
42. Fatmi, R., Rashad, S., and Integlia, R. (2019, January 7–9). Comparing ANN, SVM, and HMM based Machine Learning Methods for American Sign Language Recognition using Wearable Motion Sensors. Proceedings of the 2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.10.1109/CCWC.2019.8666491 ↗
43. Smart Wearable Hand Device for Sign Language Interpretation System With Sensors Fusion10.1109/JSEN.2017.2779466 ↗
44. Wei, S., Chen, X., Yang, X., Cao, S., and Zhang, X. (2016). A Component-Based Vocabulary-Extensible Sign Language Gesture Recognition Framework. Sensors, 16.10.3390/s16040556 ↗
45. SkinGest: Artificial skin for gesture recognition via filmy stretchable strain sensors10.1080/01691864.2018.1490666 ↗
46. Chinese Sign Language Recognition Based on an Optimized Tree-Structure Framework10.1109/JBHI.2016.2560907 ↗
47. Dawod, A.Y., and Chakpitak, N. (2019, January 26–28). Novel Technique for Isolated Sign Language Based on Fingerspelling Recognition. Proceedings of the 2019 13th International Conference on Software, Knowledge, Information Management and Applications (SKIMA), Island of Ulkulhas, Maldives.10.1109/SKIMA47702.2019.8982452 ↗
48. HGM-4: A new multi-cameras dataset for hand gesture recognition10.1016/j.dib.2020.105676 ↗
49. Hand Gesture Recognition for Thai Sign Language in Complex Background Using Fusion of Depth and Color Video10.1016/j.procs.2016.05.113 ↗
50. Jane, S.P.Y., and Sasidhar, S. (2018, January 12–15). Sign Language Interpreter: Classification of Forearm EMG and IMU Signals for Signing Exact English. Proceedings of the 2018 IEEE 14th International Conference on Control and Automation (ICCA), Anchorage, Ak, USA.10.1109/ICCA.2018.8444266 ↗
51. Flexible FTIR Spectral Imaging Enhancement for Industrial Robot Infrared Vision Sensing10.1109/TII.2019.2934728 ↗
52. Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018). Proceedings of the Computer Vision—ECCV 2018, Springer.10.1007/978-3-030-01234-2 ↗
53. Hand Gesture Recognition for Sign Language Using 3DCNN10.1109/ACCESS.2020.2990434 ↗
54. Hand gesture recognition using deep feature fusion network based on wearable sensors
55. DeepArSLR: A novel signer-independent deep learning framework for isolated arabic sign language gestures recognition10.1109/ACCESS.2020.2990699 ↗
56. Hand sign language recognition using multi-view hand skeleton10.1016/j.eswa.2020.113336 ↗
57. Gesture recognition from RGB images using convolutional neural network-attention based system10.1002/cpe.7230 ↗
58. A convolutional neural network and classical moments-based feature fusion model for gesture recognition10.1007/s00530-022-00951-5 ↗
59. CNN based feature extraction and classification for sign language10.1007/s11042-020-09829-y ↗
60. Barbhuiya, A.A., Karsh, R.K., and Dutta, S. (2021). MCCS, Proceedings of the Fifth International Conference on Microelectronics, Computing and Communication Systems, 2020, Springer.
61. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
62. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
63. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
64. Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., and Tang, X. (2017, January 21–26). Residual attention network for image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.683 ↗
65. Hu, J., Shen, L., Albanie, S., Sun, G., and Vedaldi, A. (2018, January 3–8). Gather-excite: Exploiting feature context in convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
66. Hu, J., Shen, L., and Sun, G. (2018, January 18–22). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
67. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–22). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
68. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (November, January 27). Searching for mobilenetv3. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
69. Tan, M., and Le, Q. (2019, January 10–15). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning, Long Beach, Long Beach Convention & Entertainment Center, Long Beach, CA, USA.
70. Sun, C., Shrivastava, A., Singh, S., and Gupta, A. (2017, January 22–29). Revisiting unreasonable effectiveness of data in deep learning era. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.97 ↗
71. Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z.H., Tay, F.E., Feng, J., and Yan, S. (2021, January 11–17). Tokens-to-token vit: Training vision transformers from scratch on imagenet. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00060 ↗
72. Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. (2021, January 11–17). Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00061 ↗
73. Chu, X., Tian, Z., Zhang, B., Wang, X., Wei, X., Xia, H., and Shen, C. (2021). Conditional positional encodings for vision transformers. arXiv.
74. Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., and Zhang, L. (2021, January 11–17). Cvt: Introducing convolutions to vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00009 ↗
75. Ji, Y., Kim, S., and Lee, K.B. (2017, January 10–12). Sign language learning system with image sampling and convolutional neural network. Proceedings of the 2017 First IEEE International Conference on Robotic Computing (IRC), Taichung, Taiwan.10.1109/IRC.2017.40 ↗
76. Yang, S., Jung, S., Kang, H., and Kim, C. (2020). Proceedings of the International Conference on Multimedia Modeling, Springer.
77. Shin, H., Kim, W.J., and Jang, K.a. (2019, January 23–25). Korean sign language recognition based on image and convolution neural network. Proceedings of the 2nd International Conference on Image and Graphics Processing, Singapore.10.1145/3313950.3313967 ↗
78. A deep neural framework for continuous sign language recognition by iterative training10.1109/TMM.2018.2889563 ↗
79. Kayhan, O.S., and Gemert, J.C.v. (2020, January 13–19). On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.
80. Lowe, D.G. (1999, January 20–25). Object recognition from local scale-invariant features. Proceedings of the Seventh IEEE International Conference on Computer Vision, Corfu, Greece.10.1109/ICCV.1999.790410 ↗
81. Shaw, P., Uszkoreit, J., and Vaswani, A. (2018). Self-attention with relative position representations. arXiv.10.18653/v1/N18-2074 ↗
82. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Kerkyra, Greece.10.1109/ICCV48922.2021.00986 ↗
83. Li, J., Hassani, A., Walton, S., and Shi, H. (2021). Convmlp: Hierarchical convolutional mlps for vision. arXiv.
84. Loshchilov, I., and Hutter, F. (2017). Decoupled weight decay regularization. arXiv.
85. Dozat, T. (2016, January 2–4). Incorporating Nesterov Momentum into Adam. Proceedings of the ICLR 2016 Workshop, San Juan, PR, USA.
86. Ham, S., Park, K., Jang, Y., Oh, Y., Yun, S., Yoon, S., Kim, C.J., Park, H.M., and Kweon, I.S. (2021, January 15–18). KSL-Guide: A Large-scale Korean Sign Language Dataset Including Interrogative Sentences for Guiding the Deaf and Hard-of-Hearing. Proceedings of the 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), Jodhpur, India.10.1109/FG52635.2021.9667011 ↗