Multi-Class Confidence Detection Using Deep Learning Approach
Source abstract
Abstract
The advancement of both the fields of Computer Vision (CV) and Artificial Neural Networks (ANNs) has enabled the development of effective automatic systems for analyzing human behavior. It is possible to recognize gestures, which are frequently used by people to communicate information non-verbally, by studying hand movements. So, the main contribution of this research is the collected dataset, which is taken from open-source videos of the relevant subjects that contain actions that depict confidence levels. The dataset contains high-quality frames with minimal bias and less noise. Secondly, we have chosen the domain of confidence determination during social issues such as interviews, discussions, or criminal investigations. Thirdly, the proposed model is a combination of two high-performing models, i.e., CNN (GoogLeNet) and LSTM. GoogLeNet is the state-of-the-art architecture for hand detection and gesture recognition. LSTM prevents the loss of information by keeping temporal data. So the combination of these two outperformed during the training and testing process. This study presents a method to recognize different categories of Self-Efficacy by performing multi-class classification based on the current situation of hand movements using visual data processing and feature extraction. The proposed architecture pre-processes the sequence of images collected from different scenarios, including humans, and their quality frames are extracted. These frames are then processed to extract and analyze the features regarding their body joints and hand position and classify them into four different classes related to efficacy, i.e., confidence, cooperation, confusion, and uncomfortable. The features are extracted using a combination framework of customized Convolutional Neural Network (CNN) layers with Long Short-Term Memory (LSTM) for feature extraction and classification. Remarkable results have been achieved from this study representing 90.48% accuracy with effective recognition of human body gestures through deep learning approaches.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Human Pose and Action Recognition · Gait Recognition and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, University of Engineering & Technology (UET), Lahore 54890, Pakistan
Department of Computer Science, Centro de Investigación y de Estudios Avanzados del Instituto Politécnico Nacional (CINVESTAV), Mexico City 36824, Mexico
References (55)
Show source-provided references
1. (2022, October 25). What Is Computer Vision?. Available online: https://www.ibm.com/topics/computer-vision.
2. Hand gesture classification using a novel CNN-crow search algorithm10.1007/s40747-021-00324-x ↗
3. Hand gesture recognition via enhanced densely connected convolutional neural network10.1016/j.eswa.2021.114797 ↗
4. Zhang, T. (2021). Wireless Networks, Springer.
5. Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications10.1126/sciadv.aaz8693 ↗
6. Kendon, A. (2004). Gesture: Visible Action as Utterance, University of Pennsylvania.10.1017/CBO9780511807572 ↗
7. Implementation and Analysis of AI-Based Gesticulation Control for Impaired People10.1155/2022/4656939 ↗
8. Xing, Y., and Zhu, J. (2021). Deep Learning-Based Action Recognition with 3D Skeleton: A Survey, Wiley.10.1049/cit2.12014 ↗
9. R-CNN and wavelet feature extraction for hand gesture recognition with EMG signals10.1007/s00521-020-05349-w ↗
10. Islam, M.R., Mitu, U.K., Bhuiyan, R.A., and Shin, J. (2018, January 24–27). Hand gesture feature extraction using deep convolutional neural network for recognizing American sign language. Proceedings of the IEEE 2018 4th International Conference on Frontiers of Signal Processing (ICFSP), Poitiers, France.10.1109/ICFSP.2018.8552044 ↗
11. Analysis of explainers of black box deep neural networks for computer vision: A survey10.3390/make3040048 ↗
12. Image captioning with deep bidirectional LSTMs and multi-task learning
13. Bhardwaj, S., Srinivasan, M., and Khapra, M.M. (2019, January 15–19). Efficient video classification using fewer frames. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00044 ↗
14. A spatiotemporal attention-based ResC3D model for large-scale gesture recognition10.1007/s00138-018-0996-x ↗
15. Ahuja, M.K., and Singh, A. (2015, January 1–2). Static vision based Hand Gesture recognition using principal component analysis. Proceedings of the 2015 IEEE 3rd International Conference on MOOCs, Innovation and Technology in Education (MITE), Amritsar, India.10.1109/MITE.2015.7375353 ↗
16. Oudah, M., Al-Naji, A., and Chahl, J. (2020). Hand gesture recognition based on computer vision: A review of techniques. J. Imaging, 6.10.3390/jimaging6080073 ↗
17. Visual-interactive semi-supervised labeling of human motion capture data10.2352/ISSN.2470-1173.2017.1.VDA-387 ↗
18. Zhan, F. (August, January 30). Hand gesture recognition with convolution neural networks. Proceedings of the 2019 IEEE 20th International Conference on Information Reuse and Integration for Data Science (IRI), Los Angeles, CA, USA.10.1109/IRI.2019.00054 ↗
19. Ryoo, M.S., and Aggarwal, J.K. (October, January 29). Spatio-temporal relationship match: Video structure comparison for recognition of complex human activities. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.10.1109/ICCV.2009.5459361 ↗
20. Human activity analysis: A review10.1145/1922649.1922653 ↗
21. Scovanner, P., Ali, S., and Shah, M. (2007, January 24–29). A 3-dimensional sift descriptor and its application to action recognition. Proceedings of the 15th ACM International Conference on Multimedia, Augsburg, Germany.10.1145/1291233.1291311 ↗
22. Robust human activity recognition using multimodal feature-level fusion10.1109/ACCESS.2019.2913393 ↗
23. Benitez-Garcia, G., Prudente-Tixteco, L., Castro-Madrid, L.C., Toscano-Medina, R., Olivares-Mercado, J., Sanchez-Perez, G., and Villalba, L.J.G. (2021). Improving real-time hand gesture recognition with semantic segmentation. Sensors, 21.10.3390/s21020356 ↗
24. Deep Learning Approaches for Air-writing Using Single UWB Radar10.1109/JSEN.2022.3172727 ↗
25. HCP: A Flexible CNN Framework for Multi-Label Image Classification10.1109/TPAMI.2015.2491929 ↗
26. Using multi-label classification to improve object detection10.1016/j.neucom.2019.08.089 ↗
27. Spatially Self-Paced Convolutional Networks for Change Detection in Heterogeneous Images10.1109/JSTARS.2021.3078437 ↗
28. Development of hand gesture recognition system using machine learning10.1007/s12652-020-02314-2 ↗
29. He, J., and Zhang, H. (2008, January 16–18). A real time face detection method in human-machine interaction. Proceedings of the 2008 2nd International Conference on Bioinformatics and Biomedical Engineering, Shanghai, China.10.1109/ICBBE.2008.823 ↗
30. Yang, J., Lu, W., and Waibel, A. Skin-color modeling and adaptation. Proceedings of the Asian Conference on Computer Vision.
31. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
32. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A.A. (2017, January 4–9). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the Thirty-first AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11231 ↗
33. You, J., and Korhonen, J. (2020, January 25–28). Attention boosted deep networks for video classification. Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Online.10.1109/ICIP40778.2020.9190996 ↗
34. Human action recognition using attention based LSTM network with dilated CNN features10.1016/j.future.2021.06.045 ↗
35. Khan, S., Khan, M.A., Alhaisoni, M., Tariq, U., Yong, H.S., Armghan, A., and Alenezi, F. (2021). Human action recognition: A paradigm of best deep learning features selection and serial based extended fusion. Sensors, 21.10.3390/s21237941 ↗
36. An analysis of convolutional long short-term memory recurrent neural networks for gesture recognition10.1016/j.neucom.2016.12.088 ↗
37. Xi, C., Chen, J., Zhao, C., Pei, Q., and Liu, L. (2018, January 25–27). Real-time hand tracking using kinect. Proceedings of the 2nd International Conference on Digital Signal Processing, Tokyo, Japan.10.1145/3193025.3193056 ↗
38. Konstantinidis, D., Dimitropoulos, K., and Daras, P. (2018, January 3–5). Sign language recognition based on hand and body skeletal data. Proceedings of the IEEE 2018-3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), Helsinki, Finland.10.1109/3DTV.2018.8478467 ↗
39. De Smedt, Q., Wannous, H., Vandeborre, J.P., Guerry, J., Saux, B.L., and Filliat, D. (2017, January 23–24). 3d hand gesture recognition using a depth and skeletal dataset: Shrec’17 track. Proceedings of the Workshop on 3D Object Retrieval, Graz, Austria.
40. Ren, Z., Meng, J., and Yuan, J. (2011, January 23–24). Depth camera based hand gesture recognition and its applications in human-computer-interaction. Proceedings of the IEEE 2011 8th International Conference on Information, Communications & Signal Processing, Lyon, France.
41. Sahoo, J.P., Ari, S., and Patra, S.K. (2019, January 13–16). Hand gesture recognition using PCA based deep CNN reduced features and SVM classifier. Proceedings of the 2019 IEEE International Symposium on Smart Electronic Systems (iSES)(Formerly iNiS), Singapore.10.1109/iSES47678.2019.00056 ↗
42. Kinect sensor-based long-distance hand gesture recognition and fingertip detection with depth information10.1155/2018/5809769 ↗
43. Desai, S. Segmentation and recognition of fingers using Microsoft Kinect. Proceedings of the International Conference on Communication and Networks.
44. Bakar, M.Z.A., Samad, R., Pebrianti, D., and Aan, N.L.Y. (2014, January 28–30). Real-time rotation invariant hand tracking using 3D data. Proceedings of the 2014 IEEE International Conference on Control System, Computing and Engineering (ICCSCE 2014), Penang, Malaysia.
45. Bamwenda, J., and Özerdem, M. (2019). Recognition of static hand gesture with using ANN and SVM. Dicle Univ. J. Eng.
46. Desai, S., and Desai, A. Human Computer Interaction through hand gestures for home automation using Microsoft Kinect. Proceedings of the International Conference on Communication and Networks.
47. Tekin, B., Bogo, F., and Pollefeys, M. (2019, January 15–19). H+ o: Unified egocentric recognition of 3d hand-object poses and interactions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00464 ↗
48. Wan, C., Probst, T., Gool, L.V., and Yao, A. (2019, January 15–19). Self-supervised 3d hand pose estimation through training by fitting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01111 ↗
49. Ge, L., Ren, Z., Li, Y., Xue, Z., Wang, Y., Cai, J., and Yuan, J. (2019, January 15–19). 3d hand shape and pose estimation from a single rgb image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01109 ↗
50. MEgATrack: Monochrome egocentric articulated hand-tracking for virtual reality10.1145/3386569.3392452 ↗
51. Wu, X., Finnegan, D., O’Neill, E., and Yang, Y.L. (2018, January 8–14). Handmap: Robust hand pose estimation via intermediate dense guidance map supervision. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01270-0_15 ↗
52. Alnaim, N., Abbod, M., and Albar, A. (2019, January 11–13). Hand gesture recognition using convolutional neural network for people who have experienced a stroke. Proceedings of the 2019 3rd International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), Ankara, Turkey.10.1109/ISMSIT.2019.8932739 ↗
53. Chung, H.Y., Chung, Y.L., and Tsai, W.F. (2019, January 13–15). An efficient hand gesture recognition system based on deep CNN. Proceedings of the 2019 IEEE International Conference on Industrial Technology (ICIT), Melbourne, VIC, Australia.10.1109/ICIT.2019.8755038 ↗
54. Tiny hand gesture recognition without localization via a deep convolutional network10.1109/TCE.2017.014971 ↗
55. Hand gesture recognition based on convolution neural network10.1007/s10586-017-1435-x ↗