Ensemble Learning of Multiple Deep CNNs Using Accuracy-Based Weighted Voting for ASL Recognition
Source abstract
Abstract
More than four million people worldwide suffer from hearing loss. Recently, new CNNs and deep ensemble-learning technologies have brought promising opportunities to the image-recognition field, so many studies aiming to recognize American Sign Language (ASL) have been conducted to help these people express their thoughts. This paper proposes an ASL Recognition System using Multiple deep CNNs and accuracy-based weighted voting (ARS-MA) composed of three parts: data preprocessing, feature extraction, and classification. Ensemble learning using multiple deep CNNs based on LeNet, AlexNet, VGGNet, GoogleNet, and ResNet were set up for the feature extraction and their results were used to create three new datasets for classification. The proposed accuracy-based weighted voting (AWV) algorithm and four existing machine algorithms were compared for the classification. Two parameters, α and λ, are introduced to increase the accuracy and reduce the testing time in AWV. The experimental results show that the proposed ARS-MA achieved 98.83% and 98.79% accuracy on the ASL Alphabet and ASLA datasets, respectively.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Hearing Impairment and Communication · Gait Recognition and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Engineering, Chonnam National University, Yeosu 59626, Republic of Korea
Department of Electronic Engineering, Gyeongsang National University, Jinju 52828, Republic of Korea
References (44)
Show source-provided references
1. (2022, February 01). World Health Organization (WHO). Available online: www.who.int/deafness/world-hearing-day/whd-2018/en.
2. Static Hand Gesture Recognition for American Sign Language using Deep Convolutional Neural Network
3. Technical approaches to Chinese sign language processing: A review10.1109/ACCESS.2019.2929174 ↗
4. (2022, June 07). National Institute on Deafness and Other Communication Disorders (NIDCD), Available online: https://www.nidcd.nih.gov/health/american-sign-language.
5. Sign Language Recognition: A Deep Survey10.1016/j.eswa.2020.113794 ↗
6. Automatic recognition method for the repeat size of a weave pattern on a woven fabric image10.1177/0040517518801197 ↗
7. Spatiotemporal Feature Descriptor for Micro-Expression Recognition Using Local Cube Binary Pattern10.1109/ACCESS.2019.2950339 ↗
8. Kim, J., Kim, J., Kim, H., Shim, M., and Choi, E. (2020). CNN-Based Network Intrusion Detection against Denial-of-Service Attacks. Electronics, 9.10.3390/electronics9060916 ↗
9. Real-time vernacular sign language recognition using mediapipe and machine learning
10. Chuan, C.H., Regina, E., and Guardino, C. (2014, January 3–5). American sign language recognition using leap motion sensor. Proceedings of the 2014 13th International Conference on Machine Learning and Applications, Detroit, MI, USA.10.1109/ICMLA.2014.110 ↗
11. An efficient sign language recognition (SLR) system using Camshift tracker and hidden Markov model (hmm)10.1007/s42979-021-00485-z ↗
12. Ahmed, W., Chanda, K., and Mitra, S. (2017, January 11–14). Vision based Hand Gesture Recognition using Dynamic Time Warping for Indian Sign Language. Proceedings of the 2016 international conference on information science (ICIS), Dublin, Ireland.10.1109/INFOSCI.2016.7845312 ↗
13. Hasan, M.M., Srizon, A.Y., Sayeed, A., and Hasan, M.A.M. (2020, January 28–29). Classification of sign language characters by applying a deep convolutional neural network. Proceedings of the 2020 2nd International Conference on Advanced Information and Communication Technology (ICAICT), Dhaka, Bangladesh.10.1109/ICAICT51780.2020.9333456 ↗
14. Pigou, L., Dieleman, S., and Kindermans, P.J. (2014). Sign language recognition using convolutional neural networks. European Conference on Computer Vision, Springer.10.1007/978-3-319-16178-5_40 ↗
15. Jing, L., Vahdani, E., Huenerfauth, M., and Tian, Y. (2019). Recognizing American sign language manual signs from RGB-D videos. arXiv.
16. Huang, J., Zhou, W., Li, H., and Li, W. (July, January 29). Sign language recognition using 3d convolutional neural networks. Proceedings of the 2015 IEEE International Conference on Multimedia and Expo (ICME), Torino, Italy.
17. Incorporating Relative Position Information in Transformer-Based Sign Language Recognition and Translation10.1109/ACCESS.2021.3122921 ↗
18. De Coster, M., Van Herreweghe, M., and Dambre, J. (2020, January 11–16). European Language Resources Association (ELRA). Sign language recognition with transformer networks. Proceedings of the 12th International Conference on Language Resources and Evaluation, Palais du Pharo, France.
19. Full Transformer Network with Masking Future for Word-Level Sign Language Recognition10.1016/j.neucom.2022.05.051 ↗
20. Ye, Y., Tian, Y., Huenerfauth, M., and Liu, J. (2018, January 18–22). Recognizing american sign language gestures from within continuous videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00280 ↗
21. Deep Ensemble Learning for Human Action Recognition in Still Images10.1155/2020/6352021 ↗
22. Zaidi, S., Zela, A., Elsken, T., Holmes, C., Hutter, F., and Teh, Y.W. (2021). Neural ensemble search for uncertainty estimation and dataset shift. Advances in Neural Information Processing Systems, Morgan Kaufmann Publishers Inc.
23. Hao, A., Min, Y., and Chen, X. (2021, January 10–17). Self-mutual distillation learning for continuous sign language recognition. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.01111 ↗
24. A Comprehensive Study on Deep Learning-based Methods for Sign Language Recognition10.1109/TMM.2021.3070438 ↗
25. Kothadiya, D., Bhatt, C., Sapariya, K., Patel, K., Gil-González, A.-B., and Corchado, J.M. (2022). Deepsign: Sign Language Detection and Recognition Using Deep Learning. Electronics, 11.10.3390/electronics11111780 ↗
26. Kania, K., and Markowska-Kaczmar, U. (2018). American Sign Language Fingerspelling Recognition Using Wide Residual Networks. International Conference on Artificial Intelligence and Soft Computing, Springer.10.1007/978-3-319-91253-0_10 ↗
27. Bousbai, K., and Merah, M. (2019, January 24–25). A Comparative Study of Hand Gestures Recognition Based on MobileNetV2 and ConvNet Models. Proceedings of the 2019 6th International Conference on Image and Signal Processing and their Applications (ISPA), Mostaganem, Algeria.10.1109/ISPA48434.2019.8966918 ↗
28. Compact joints encoding for skeleton-based dynamic hand gesture recognition10.1016/j.cag.2021.04.017 ↗
29. Ma, Y., Xu, T., and Kim, K. (2022). Two-Stream Mixed Convolutional Neural Network for American Sign Language Recognition. Sensors, 22.10.3390/s22165959 ↗
30. A survey on ensemble learning10.1007/s11704-019-8208-z ↗
31. Hrúz, M., Gruber, I., Kanis, J., Boháček, M., Hlaváč, M., and Krňoul, Z. (2022). One Model is Not Enough: Ensembles for Isolated Sign Language Recognition. Sensors, 22.10.3390/s22135043 ↗
32. Ensemble Learners of Multiple Deep CNNs for Pulmonary Nodules Classification Using CT Images10.1109/ACCESS.2019.2933670 ↗
33. (2021, February 27). ASL Alphabet Dataset. Available online: https://www.kaggle.com/datasets/grassknoted/asl-alphabet.
34. (2021, February 27). ASLA Dataset. Available online: https://www.kaggle.com/datasets/debashishsau/aslamerican-sign-language-aplhabet-dataset.
35. Machine Learning Models for Predicting Hearing Prognosis in Unilateral Idiopathic Sudden Sensorineural Hearing Loss10.21053/ceo.2019.01858 ↗
36. Karlos, S., Kostopoulos, G., and Kotsiantis, S. (2020). A soft-voting ensemble based co-training scheme using static selection for binary classification problems. Algorithms, 13.10.3390/a13010026 ↗
37. Yanmei, H., Bo, W., and Zhaomin, Z. (2020). An improved LeNet-5 model for Image Recognition. Proceedings of the 2020 4th International Conference on Electronic Information Technology and Computer Engineering, Association for Computing Machinery.10.1145/3443467.3443797 ↗
38. Image Classification Algorithm Based on Improved AlexNet10.1088/1742-6596/1813/1/012051 ↗
39. (2020, January 12–14). Zhiqi, Yang. Gesture recognition based on improved VGGNET convolutional neural network. Proceedings of the IEEE 5th Information Technology and Mechatronics Engineering Conference (ITOEC), Chongqing, China.
40. Variations of AlexNet and GoogLeNet to Improve Korean Character Recognition Performance
41. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
42. How does batch normalization help optimization?
43. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions10.1186/s40537-021-00444-8 ↗
44. When to consult precision-recall curves10.1177/1536867X20909693 ↗