Real-Time Hand Gesture Recognition Based on Deep Learning YOLOv3 Model
Source abstract
Abstract
Using gestures can help people with certain disabilities in communicating with other people. This paper proposes a lightweight model based on YOLO (You Only Look Once) v3 and DarkNet-53 convolutional neural networks for gesture recognition without additional preprocessing, image filtering, and enhancement of images. The proposed model achieved high accuracy even in a complex environment, and it successfully detected gestures even in low-resolution picture mode. The proposed model was evaluated on a labeled dataset of hand gestures in both Pascal VOC and YOLO format. We achieved better results by extracting features from the hand and recognized hand gestures of our proposed YOLOv3 based model with accuracy, precision, recall, and an F-1 score of 97.68, 94.88, 98.66, and 96.70%, respectively. Further, we compared our model with Single Shot Detector (SSD) and Visual Geometry Group (VGG16), which achieved an accuracy between 82 and 85%. The trained model can be used for real-time detection, both for static hand images and dynamic gestures recorded on a video.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Human Pose and Action Recognition · Hearing Impairment and Communication
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, University of Management and Technology, Lahore 54770, Pakistan
Department of Software Engineering, University of Management and Technology, Lahore 54770, Pakistan
Department of Computer Engineering, National University of Technology, Islamabad 44000, Pakistan
Information Systems Department, College of Computer Science and Information Technology, University of Anbar, Anbar 31001, Iraq
Faculty of Applied Mathematics, Silesian University of Technology, 44-100 Gliwice, Poland
Department of Applied Informatics, Vytautas Magnus University, 44404 Kaunas, Lithuania
College of Agriculture, Al-Muthanna University, Samawah 66001, Iraq
References (63)
Show source-provided references
1. Fang, Y., Wang, K., Cheng, J., and Lu, H. (2007, January 2–5). A Real-Time Hand Gesture Recognition Method. Proceedings of the Multimedia and Expo, IEEE International Conference On Multimedia and Expo, Beijing, China.10.1109/ICME.2007.4284820 ↗
2. Oudah, M., Al-Naji, A., and Chahl, J. (2020). Hand Gesture Recognition Based on Computer Vision: A Review of Techniques. J. Imaging, 6.10.3390/jimaging6080073 ↗
3. Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation10.1109/ACCESS.2020.3032140 ↗
4. Vaitkevičius, A., Taroza, M., Blažauskas, T., Damaševičius, R., Maskeliūnas, R., and Woźniak, M. (2019). Recognition of american sign language gestures in a virtual reality using leap motion. Appl. Sci., 9.10.3390/app9030445 ↗
5. Rezende, T.M., Almeida, S.G.M., and Guimarães, F.G. (2021). Development and validation of a brazilian sign language database for human gesture recognition. Neural Comput. Appl.10.1007/s00521-021-05802-4 ↗
6. Recognition of basketball referee signals from real-time videos10.1007/s12652-019-01209-1 ↗
7. A framework of human action recognition using length control features fusion and weighted entropy-variances based feature selection10.1016/j.imavis.2020.104090 ↗
8. Facial feature extraction and pose determination10.1016/S0031-3203(99)00176-4 ↗
9. Detection of sitting posture using hierarchical image composition and deep learning10.7717/peerj-cs.442 ↗
10. Multiple kinect based system to monitor and analyze key performance indicators of physical training10.1186/s13673-020-00256-4 ↗
11. An ANN-based gesture recognition algorithm for smart-home applications
12. Abraham, L., Urru, A., Normani, N., Wilk, M.P., Walsh, M., and O’Flynn, B. (2018). Hand Tracking and Gesture Recognition Using Lensless Smart Sensors. Sensors, 18.10.3390/s18092834 ↗
13. Ahmed, S., and Cho, S.H. (2020). Hand Gesture Recognition Using an IR-UWB Radar with an Inception Module-Based Classifier. Sensors, 20.10.3390/s20020564 ↗
14. Lee, D.-H., and Kwang-Seok Hong, K.-S. (December, January 30). Game interface using hand gesture recognition. Proceedings of the 5th International Conference on Computer Sciences and Convergence Information Technology, Seoul, Korea.
15. On the efficiency of a VR hand gesture-based interface for 3D object manipulations in conceptual design10.1080/10447318.2017.1296074 ↗
16. Immersive gesture interfaces for navigation of 3D maps in HMD-based mobile virtual environments
17. Del Rio Guerra, M.S., Martin-Gutierrez, J., Acevedo, R., and Salinas, S. (2019). Hand gestures in virtual and augmented 3D environments for down syndrome users. Appl. Sci., 9.10.3390/app9132641 ↗
18. Toward an unsupervised approach for daily gesture recognition in assisted living applications10.1109/JSEN.2017.2764323 ↗
19. An easily customized gesture recognizer for assisted living using commodity mobile devices10.1155/2018/3180652 ↗
20. PRAXIS: Towards automatic cognitive assessment using gesture recognition10.1016/j.eswa.2018.03.063 ↗
21. Kaczmarek, W., Panasiuk, J., Borys, S., and Banach, P. (2020). Industrial robot control by means of gestures and voice commands in off-line and on-line mode. Sensors, 20.10.20944/preprints202010.0455.v1 ↗
22. Gesture-based human-robot interaction for human assistance in manufacturing10.1007/s00170-018-2788-x ↗
23. Young, G., Milne, H., Griffiths, D., Padfield, E., Blenkinsopp, R., and Georgiou, O. (2020, January 25–30). Designing mid-air haptic gesture controlled user interfaces for cars. Proceedings of the ACM on Human-Computer Interaction, 4(EICS), Article No. 81, Honolulu, HI, USA.10.1145/3397869 ↗
24. A novel hand gesture recognition method based on 2-channel sEMG10.3233/THC-174567 ↗
25. Zhao, L., and Li, S. (2020). Object detection algorithm based on improved YOLOv3. Electronics, 9.10.3390/electronics9030537 ↗
26. Kulikajevas, A., Maskeliūnas, R., Damaševičius, R., and Ho, E.S.L. (2020). 3D object reconstruction from imperfect depth data using extended yolov3 network. Sensors, 20.10.3390/s20072025 ↗
27. Ni, Z., Chen, J., Sang, N., Gao, C., and Liu, L. (2018, January 7–10). Light YOLO for High-Speed Gesture Recognition. Proceedings of the 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.10.1109/ICIP.2018.8451766 ↗
28. Chen, L., Fu, J., Wu, Y., Li, H., and Zheng, B. (2020). Hand Gesture Recognition Using Compact CNN via Surface Electromyography Signals. Sensors, 20.10.3390/s20030672 ↗
29. Colli-Alfaro, J.G., Ibrahim, A., and Trejos, A.L. (2019, January 24–28). Design of User-Independent Hand Gesture Recognition Using Multilayer Perceptron Networks and Sensor Fusion Techniques. Proceedings of the IEEE 16th International Conference on Rehabilitation Robotics (ICORR), Toronto, ON, Canada.10.1109/ICORR.2019.8779533 ↗
30. A hidden markov model-based isolated and meaningful hand gesture recognition
31. Nyirarugira, C., Choi, H.-R., Kim, J., Hayes, M., and Kim, T. (2013, January 16–18). Modified levenshtein distance for real-time gesture recognition. Proceedings of the 6th International Congress on Image and Signal Processing (CISP), Hangzhou, China.10.1109/CISP.2013.6745306 ↗
32. Albawi, S., Bayat, O., Al-Azawi, S., and Ucan, O.N. (2018). Social Touch Gesture Recognition Using Convolutional Neural Network. Comput. Intell. Neurosci., 1–10.10.1155/2018/6973103 ↗
33. Ju, M., Luo, H., Wang, Z., Hui, B., and Chang, Z. (2019). The Application of Improved YOLO V3 in Multi-Scale Target Detection. Appl. Sci., 9.10.3390/app9183775 ↗
34. Intelligent dynamic gesture recognition using CNN empowered by edit distance
35. Hand gesture recognition for sign language using 3DCNN10.1109/ACCESS.2020.2990434 ↗
36. Do, N., Kim, S., Yang, H., and Lee, G. (2020). Robust hand shape features for dynamic hand gesture recognition using multi-level feature LSTM. Appl. Sci., 10.10.3390/app10186293 ↗
37. Elboushaki, A., Hannane, R., Afdel, K., and Koutti, L. (2020). MultiD-CNN: A multi-dimensional feature learning approach based on deep convolutional networks for gesture recognition in RGB-D image sequences. Expert Syst. Appl., 139.10.1016/j.eswa.2019.112829 ↗
38. Dynamic gesture recognition based on feature fusion network and variant ConvLSTM10.1049/iet-ipr.2019.1248 ↗
39. Tan, Y.S., Lim, K.M., and Lee, C.P. (2021). Hand gesture recognition via enhanced densely connected convolutional neural network. Expert Syst. Appl., 175.10.1016/j.eswa.2021.114797 ↗
40. Tran, D., Ho, N., Yang, H., Baek, E., Kim, S., and Lee, G. (2020). Real-time hand gesture spotting and recognition using RGB-D camera and 3D convolutional neural network. Appl. Sci., 10.10.3390/app10020722 ↗
41. Rahim, M.A., Islam, M.R., and Shin, J. (2019). Non-Touch Sign Word Recognition Based on Dynamic Hand Gesture Using Hybrid Segmentation and CNN Feature Fusion. Appl. Sci., 9.10.3390/app9183790 ↗
42. Mambou, S., Krejcar, O., Maresova, P., Selamat, A., and Kuca, K. (2019). Novel Hand Gesture Alert System. Appl. Sci., 9.10.3390/app9163419 ↗
43. Ashiquzzaman, A., Lee, H., Kim, K., Kim, H.-Y., Park, J., and Kim, J. (2020). Compact Spatial Pyramid Pooling Deep Convolutional Neural Network Based Hand Gestures Decoder. Appl. Sci., 10.10.3390/app10217898 ↗
44. Benitez-Garcia, G., Prudente-Tixteco, L., Castro-Madrid, L.C., Toscano-Medina, R., Olivares-Mercado, J., Sanchez-Perez, G., and Villalba, L.J.G. (2021). Improving Real-Time Hand Gesture Recognition with Semantic Segmentation. Sensors, 21.10.3390/s21020356 ↗
45. The OpenCV Library
46. The pascal visual object classes (voc) challenge10.1007/s11263-009-0275-4 ↗
47. Redmon, J., and Farhadi, A. (2018). YOLOv3: An Incremental Improvement. arXiv.
48. Qassim, H., Verma, A., and Feinzimer, D. (2018, January 8–10). Compressed residual-VGG16 CNN model for big data places image recognition. Proceedings of the 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA.10.1109/CCWC.2018.8301729 ↗
49. Fu, C., Liu, W., Ranga, A., Tyagi, A., and Berg, A.C. (2017). DSSD: Deconvolutional Single Shot Detector. arXiv.
50. Gesture Recognition from Data Streams of Human Motion Sensor Using Accelerated PSO Swarm Search Feature Selection Algorithm10.1155/2015/205707 ↗
51. Multiscale Convolutional Neural Networks for Hand Detection
52. Robust Part-Based Hand Gesture Recognition Using Kinect Sensor10.1109/TMM.2013.2246148 ↗
53. Pedoeem, J., and Huang, R. (2018, January 10–13). YOLO-LITE: A Real-Time Object Detection Algorithm Optimized for Non-GPU Computers. Proceedings of the IEEE International Conference on Big Data (Big Data), Seattle, WA, USA.
54. Sismananda, P., Abdurohman, M., and Putrada, A.G. (2020, January 4–5). Performance Comparison of Yolo-Lite and YoloV3 Using Raspberry Pi and MotionEyeOS. Proceedings of the 8th International Conference on Information and Communication Technology (ICoICT), Yogyakarta, Indonesia.10.1109/ICoICT49345.2020.9166199 ↗
55. Human-machine interaction in intelligent technologies using the augmented reality
56. Recognition of basketball referee signals from videos using histogram of oriented gradients (HOG) and support vector machine (SVM)10.1016/j.procs.2018.04.095 ↗
57. Body pose prediction based on motion sensor data and recurrent neural network10.1109/TII.2020.3015934 ↗
58. Maskeliunas, R., Damaševicius, R., and Segal, S. (2019). A review of internet of things technologies for ambient assisted living environments. Future Internet, 11.10.3390/fi11120259 ↗
59. Zhao, H., Zhou, Y., Zhang, L., Peng, Y., Hu, X., Peng, H., and Cai, X. (2020). Mixed YOLOv3-LITE: A Lightweight Real-Time Object Detection Method. Sensors, 20.10.3390/s20071861 ↗
60. Awan, M.J., Rahim, M.S.M., Salim, N., Mohammed, M.A., Garcia-Zapirain, B., and Abdulkareem, K.H. (2021). Efficient Detection of Knee Anterior Cruciate Ligament from Magnetic Resonance Imaging Using Deep Learning Approach. Diagnostics, 11.10.3390/diagnostics11010105 ↗
61. Mastoi, Q., Memon, M.S., Lakhan, A., Mohammed, M.A., Qabulio, M., Al-Turjman, F., and Abdulkareem, K.H. (2021). Machine learning-data mining integrated approach for premature ventricular contraction prediction. Neural Comput. Appl.10.1007/s00521-021-05820-2 ↗
62. Mohammed, M.A., Abdulkareem, K.H., Mostafa, S.A., Ghani, M.K.A., Maashi, M.S., Garcia-Zapirain, B., Oleagordia, I., Alhakami, H., and Al-Dhief, F.T. (2020). Voice pathology detection and classification using convolutional neural network model. Appl. Sci., 10.10.3390/app10113723 ↗
63. Review of Data Fusion Methods for Real-Time and Multi-Sensor Traffic Flow Analysis10.1109/ACCESS.2021.3069770 ↗