Gesture Detection and Recognition Based on Object Detection in Complex Background
Source abstract
Abstract
In practical human–computer interaction, a hand gesture recognition method based on improved YOLOv5 is proposed to address the problem of low recognition accuracy and slow speed with complex backgrounds. By replacing the CSP1_x module in the YOLOv5 backbone network with an efficient layer aggregation network, a richer combination of gradient paths can be obtained to improve the network’s learning and expressive capabilities and enhance recognition speed. The CBAM attention mechanism is introduced to filtering gesture features in channel and spatial dimensions, reducing various types of interference in complex background gesture images and enhancing the network’s robustness against complex backgrounds. Experimental verification was conducted on two complex background gesture datasets, EgoHands and TinyHGR, with recognition accuracies of mAP0.5:0.95 at 75.6% and 66.8%, respectively, and a recognition speed of 64 FPS for 640 × 640 input images. The results show that the proposed method can recognize gestures quickly and accurately with complex backgrounds, and has higher recognition accuracy and stronger robustness compared to YOLOv5l, YOLOv7, and other comparative algorithms.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Human Pose and Action Recognition · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Chongqing Engineering Laboratory for Transportation Engineering Application Robot, Chongqing Jiaotong University, Chongqing 400074, China
Funding
Chongqing Jiaotong University
References (55)
Show source-provided references
1. Human-machine interaction sensing technology based on hand gesture recognition: A review10.1109/THMS.2021.3086003 ↗
2. Ahmed, S., Kallu, K.D., Ahmed, S., and Cho, S.H. (2021). Hand Gestures Recognition Using Radar Sensors for Human-Computer-Interaction: A Review. Remote Sens., 13.10.3390/rs13030527 ↗
3. An empirical evaluation of two natural hand interaction systems in augmented reality10.1007/s11042-022-12864-6 ↗
4. Design of hand gesture recognition system for human-computer interaction10.1007/s11042-019-08274-w ↗
5. Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human–Robot Interaction10.1109/JSEN.2021.3059685 ↗
6. Occlusion gesture recognition based on improved SSD10.1002/cpe.6063 ↗
7. Vision-based hand gesture recognition using deep learning for the interpretation of sign language10.1016/j.eswa.2021.115657 ↗
8. Development of hand gesture recognition system using machine learning10.1007/s12652-020-02314-2 ↗
9. Design and development of a vision-based system for detection, tracking and recognition of isolated dynamic bare hand gesticulated characters10.1111/exsy.12970 ↗
10. A Novel Illumination-Robust Hand Gesture Recognition System with Event-Based Neuromorphic Vision Sensor10.1109/TASE.2020.3045880 ↗
11. Hand Gesture Recognition Enhancement Based on Spatial Fuzzy Matching in Leap Motion10.1109/TII.2019.2931140 ↗
12. A lightweight hand gesture recognition in complex backgrounds10.1016/j.displa.2022.102226 ↗
13. Hand gesture recognition via image processing techniques and deep CNN10.3233/JIFS-200385 ↗
14. Hand gesture recognition from 2D images by using convolutional capsule neural networks10.1007/s13369-021-05867-2 ↗
15. Li, J., Li, C., Han, J., Shi, Y., Bian, G., and Zhou, S. (2022). Robust Hand Gesture Recognition Using HOG-9ULBP Features and SVM Model. Electronics, 11.10.3390/electronics11070988 ↗
16. Literature review of vision-based dynamic gesture recognition using deep learning techniques10.1002/cpe.7159 ↗
17. Deep Learning Based Hand Gesture Recognition and UAV Flight Controls10.1007/s11633-019-1194-7 ↗
18. Dynamic Hand Gesture Recognition Based on Signals from Specialized Data Glove and Deep Learning Algorithms
19. Deep Learning-Based Approach for Sign Language Gesture Recognition with Efficient Hand Gesture Representation10.1109/ACCESS.2020.3032140 ↗
20. Vision-based hand signal recognition in construction: A feasibility study10.1016/j.autcon.2021.103625 ↗
21. Towards wide-scale continuous gesture recognition model for in-depth and grayscale input videos10.1007/s13042-020-01227-y ↗
22. Deep signature-based isolated and large scale continuous gesture recognition approach
23. ChaLearn Looking at People: IsoGD and ConGD Large-Scale RGB-D Gesture Recognition10.1109/TCYB.2020.3012092 ↗
24. Robust human gesture recognition by leveraging multi-scale feature fusion10.1016/j.image.2019.115768 ↗
25. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 11–14). SSD: Single shot multibox detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
26. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
27. Redmon, J., and Farhadi, A. (2017, January 21–26). YOLO9000: Better, faster, stronger. Proceedings of the Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.690 ↗
28. Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv.
29. Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv.
30. RefineDet++: Single-shot refinement neural network for object detection10.1109/TCSVT.2020.2986402 ↗
31. MAT: Multianchor Visual Tracking with Selective Search Region10.1109/TCYB.2020.3039341 ↗
32. S-RPN: Sampling-balanced region proposal network for small crop pest detection10.1016/j.compag.2021.106290 ↗
33. Light invariant real-time robust hand gesture recognition10.1016/j.ijleo.2017.11.158 ↗
34. Gesture recognition by combining gesture principal direction and Hausdorff-like distance
35. Gesture recognition method combining dense convolution and spatial transformation network
36. Xu, C., Cai, W., Li, Y., Zhou, J., and Wei, L. (2019). Accurate Hand Detection from Single-Color Images by Reconstructing Hand Appearances. Sensors, 20.10.3390/s20010192 ↗
37. Generative Adversarial Networks: An Overview10.1109/MSP.2017.2765202 ↗
38. Soe, H.M., and Naing, T.M. Real-time hand pose recognition using faster region-based convolutional neural network. Proceedings of the First International Conference on Big Data Analysis and Deep Learning.
39. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster r-cnn: Towards real-time object detection with region proposal networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
40. Kiwifruit detection in field images using Faster R-CNN with ZFNet10.1016/j.ifacol.2018.08.059 ↗
41. Attention based detection and recognition of hand postures against complex backgrounds10.1007/s11263-012-0560-5 ↗
42. Gesture recognition based on YOLO algorithm
43. Static gesture real-time recognition method based on ShuffleNetv2-YOLOv3 model
44. Ma, N., Zhang, X., Zheng, H.T., and Sun, J. (2018, January 8–14). Shufflenet v2: Practical guidelines for efficient cnn architecture design. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_8 ↗
45. Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8–14). CBAM: Convolutional block attention module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
46. Gesture Recognition Based on Improved YOLOv4-tiny Algorithm
47. Identification and Classification of Mechanical Damage During Continuous Harvesting of Root Crops Using Computer Vision Methods10.1109/ACCESS.2022.3157619 ↗
48. Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition10.1109/TPAMI.2015.2389824 ↗
49. Tiny hand gesture recognition without localization via a deep convolutional network10.1109/TCE.2017.014971 ↗
50. Sewing gesture recognition based on improved YOLO deep convolutional neural network
51. Hand Gesture Recognition against Complex Background Based on Deep Learning
52. Wang, C.Y., Liao, H.Y.M., and Yeh, I.H. (2022). Designing Network Design Strategies Through Gradient Path Analysis. arXiv.
53. Bambach, S., Lee, S., Crandall, D.J., and Yu, C. (2015, January 7–13). Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. Proceedings of the International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.226 ↗
54. Wang, C.Y., Bochkovskiy, A., and Liao, H.Y.M. (2022). YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv.
55. Lee, Y., and Park, J. (2020, January 13–19). Centermask: Real-time anchor-free instance segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01392 ↗