Evaluation of Benchmark Datasets and Deep Learning Models with Pre-Trained Weights for Vision-Based Dynamic Hand Gesture Recognition
Source abstract
Abstract
The integration of dynamic hand gesture recognition in computer vision-based systems promises enhanced human–computer interaction, providing a natural and intuitive way of communicating. However, achieving real-time performance efficiency is a highly challenging task. As the effectiveness of dynamic hand gesture recognition is dependent on the nature of the underlying datasets and deep learning models, selecting a diverse and effective dataset and a deep learning model is crucial to achieve reliable performance. This study explores the effectiveness of benchmark hand gesture recognition datasets in training lightweight deep learning models for robust performance. The objective is to evaluate and analyze these datasets and models through training and evaluation for use in practical applications. For the evaluation of these datasets and models, we analyze the models’ performances by evaluation metrics, such as precision, recall, F1-score, specificity, and accuracy. For an unbiased comparison, both subjective and objective metrics are reported, thus offering significant insights on understanding dataset–model interactions in hand gesture recognition.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Robotics and Automated Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronics Engineering, Sejong University, Seoul 05006, Republic of Korea
Department of Electrical Engineering, Sejong University, Seoul 05006, Republic of Korea
Funding
Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government (MSIT) · Unmanned Vehicles Core Technology Research and Development Program through the National Research Foundation of Korea (NRF) and the Unmanned Vehicle Advanced Research Center(UVARC) funded by the Ministry of Science and ICT, the Republic of Korea
References (51)
Show source-provided references
1. A transfer learning evaluation of deep neural networks for image classification10.3390/make4010002 ↗
2. Garg, M., Ghosh, D., and Pradhan, P.M. (2024, January 16–22). GestFormer: Multiscale Wavelet Pooling Transformer Network for Dynamic Hand Gesture Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPRW63382.2024.00254 ↗
3. Khatri, S., Chourasia, L., Jain, A., and Barge, Y. (2024, January 3–4). Touchless Control: Hand-Based Gesture Recognition for Human-Computer Interaction. Proceedings of the 2024 Parul International Conference on Engineering and Technology (PICET), Gujarat, India.10.1109/PICET60765.2024.10716196 ↗
4. Oudah, M., Al-Naji, A., and Chahl, J. (2020). Hand gesture recognition based on computer vision: A review of techniques. J. Imaging, 6.10.3390/jimaging6080073 ↗
5. An Exploration into Human–Computer Interaction: Hand Gesture Recognition Management in a Challenging Environment10.1007/s42979-023-01751-y ↗
6. A Novel Hybrid Deep Learning Architecture for Dynamic Hand Gesture Recognition10.1109/ACCESS.2024.3365274 ↗
7. Dynamic hand gesture recognition using multi-branch attention based graph and general deep learning model10.1109/ACCESS.2023.3235368 ↗
8. Survey on vision-based dynamic hand gesture recognition10.1007/s00371-023-03160-x ↗
9. Leins, D.P., Gibas, C., Brück, R., and Haschke, R. (2021). Toward more robust hand gesture recognition on EIT data. Front. Neurorobot., 15.10.3389/fnbot.2021.659311 ↗
10. Hand gesture recognition from 2D images by using convolutional capsule neural networks10.1007/s13369-021-05867-2 ↗
11. Diverse hand gesture recognition dataset10.1007/s11042-023-17268-8 ↗
12. SHAPE: A dataset for hand gesture recognition10.1007/s00521-022-07651-1 ↗
13. A Systematic Review of Hand Gesture Recognition: An Update From 2018 to 202410.1109/ACCESS.2024.3421992 ↗
14. Lee, J.W., and Yu, K.H. (2023). Wearable Drone Controller: Machine Learning-Based Hand Gesture Recognition and Vibrotactile Feedback. Sensors, 23.10.3390/s23052666 ↗
15. Reddy, A.V., Shah, K., Paul, W., Mocharla, R., Hoffman, J., Katyal, K.D., Manocha, D., De Melo, C.M., and Chellappa, R. (June, January 29). Synthetic-to-real domain adaptation for action recognition: A dataset and baseline performances. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), London, UK.10.1109/ICRA48891.2023.10160416 ↗
16. Schak, M., and Gepperth, A. (2022, January 6–9). Gesture MNIST: A new free-hand gesture dataset. Proceedings of the International Conference on Artificial Neural Networks, Bristol, UK.10.1007/978-3-031-15937-4_55 ↗
17. Multi-level feature disentanglement network for cross-dataset face forgery detection10.1016/j.imavis.2023.104686 ↗
18. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2017, January 4–9). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11231 ↗
19. Huang, T., Huang, L., You, S., Wang, F., Qian, C., and Xu, C. (2022). Lightvit: Towards light-weight convolution-free vision transformers. arXiv.
20. Tan, M., and Le, Q.V. (2019, January 9–15). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA.
21. Maaz, M., Shaker, A., Cholakkal, H., Khan, S., Zamir, S.W., Anwer, R.M., and Shahbaz Khan, F. (2022, January 23–27). Edgenext: Efficiently amalgamated cnn-transformer architecture for mobile vision applications. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-25082-8_1 ↗
22. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
23. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (November, January 27). Searching for mobilenetv3. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
24. Artificial intelligence-based human gesture tracking control techniques of Tello EDU Quadrotor Drone
25. Bello, H., Suh, S., Geißler, D., Ray, L.S.S., Zhou, B., and Lukowicz, P. (2023, January 8–12). CaptAinGlove: Capacitive and inertial fusion-based glove for real-time on edge hand gesture recognition for drone control. Proceedings of the Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing, Cancun, Mexico.10.1145/3594739.3610713 ↗
26. Althnian, A., AlSaeed, D., Al-Baity, H., Samha, A., Dris, A.B., Alzakari, N., Abou Elwafa, A., and Kurdi, H. (2021). Impact of dataset size on classification performance: An empirical evaluation in the medical domain. Appl. Sci., 11.10.3390/app11020796 ↗
27. Li, Z., Bampis, C.G., Krasula, L., Janowski, L., and Katsavounidis, I. (2020). A simple model for subject behavior in subjective experiments. arXiv.10.2352/ISSN.2470-1173.2020.11.HVEI-131 ↗
28. Effects of Visual Realism on Avatar Perception in Immersive and Non-Immersive Virtual Environments
29. Enhancing the Capabilities of Remotely Piloted Aerial Systems Through Object Detection, Face Tracking, Digital Mapping and Gesture Control
30. Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation10.1109/ACCESS.2020.3032140 ↗
31. Zulpukharkyzy Zholshiyeva, L., Kokenovna Zhukabayeva, T., Turaev, S., Aimambetovna Berdiyeva, M., and Tokhtasynovna Jambulova, D. (2021, January 11–13). Hand gesture recognition methods and applications: A literature survey. Proceedings of the 7th International Conference on Engineering & MIS 2021, Almaty, Kazakhstan.10.1145/3492547.3492578 ↗
32. Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.L., Yong, M.G., and Lee, J. (2019). Mediapipe: A framework for building perception pipelines. arXiv.
33. Openpose: Realtime multi-person 2d pose estimation using part affinity fields10.1109/TPAMI.2019.2929257 ↗
34. End-side gesture recognition method for UAV control10.1109/JSEN.2022.3218829 ↗
35. Diliberti, N., Peng, C., Kaufman, C., Dong, Y., and Hansberger, J.T. (2019, January 21–25). Real-time gesture recognition using 3D sensory data and a light convolutional neural network. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3350958 ↗
36. SimplyMime: A Dynamic Gesture Recognition and Authentication System for Smart Remote Control10.1109/JSEN.2024.3487070 ↗
37. Emerging wearable interfaces and algorithms for hand gesture recognition: A survey10.1109/RBME.2021.3078190 ↗
38. Egogesture: A new dataset and benchmark for egocentric hand gesture recognition10.1109/TMM.2018.2808769 ↗
39. Molchanov, P., Yang, X., Gupta, S., Kim, K., Tyree, S., and Kautz, J. (2016, January 27–30). Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.456 ↗
40. Materzynska, J., Berger, G., Bax, I., and Memisevic, R. (2019, January 27–28). The jester dataset: A large-scale video dataset of human gestures. Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Republic of Korea.10.1109/ICCVW.2019.00349 ↗
41. Tsai, T.H., and Tsai, Y.R. (2021). Architecture Design and VLSI Implementation of 3D Hand Gesture Recognition System. Sensors, 21.10.3390/s21206724 ↗
42. (2023, December 12). 2024. Available online: https://open.selectstar.ai/ko/gist.
43. Nemec, J., and Alba-Flores, R. (2022, January 1–4). Unmanned aerial vehicle control using hand gestures and neural networks. Proceedings of the 2022 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS), Toronto, ON, Canada.10.1109/IEMTRONICS55184.2022.9795780 ↗
44. Padhi, P., and Das, M. (2022, January 13–15). Hand Gesture Recognition using DenseNet201-Mediapipe Hybrid Modelling. Proceedings of the 2022 International Conference on Automation, Computing and Renewable Systems (ICACRS), Pudukkottai, India.10.1109/ICACRS55517.2022.10029038 ↗
45. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
46. Navigation control of drone using hand gesture based on complementary filter algorithm10.1088/1742-6596/1912/1/012034 ↗
47. Investigating bias in deep face analysis: The kanface dataset and empirical study10.1016/j.imavis.2020.103954 ↗
48. Kaczmarek, P., Mańkowski, T., and Tomczyński, J. (2019). putEMG—A surface electromyography hand gesture recognition dataset. Sensors, 19.10.3390/s19163548 ↗
49. Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 18–22). Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00716 ↗
50. Uboweja, E., Tian, D., Wang, Q., Kuo, Y.C., Zou, J., Wang, L., Sung, G., and Grundmann, M. (2023, January 2–3). On-device Real-time Custom Hand Gesture Recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCVW60793.2023.00461 ↗
51. Gesture recognition: A survey10.1109/TSMCC.2007.893280 ↗