3D Skeletal Joints-Based Hand Gesture Spotting and Classification
Source abstract
Abstract
This paper presents a novel approach to continuous dynamic hand gesture recognition. Our approach contains two main modules: gesture spotting and gesture classification. Firstly, the gesture spotting module pre-segments the video sequence with continuous gestures into isolated gestures. Secondly, the gesture classification module identifies the segmented gestures. In the gesture spotting module, the motion of the hand palm and fingers are fed into the Bidirectional Long Short-Term Memory (Bi-LSTM) network for gesture spotting. In the gesture classification module, three residual 3D Convolution Neural Networks based on ResNet architectures (3D_ResNet) and one Long Short-Term Memory (LSTM) network are combined to efficiently utilize the multiple data channels such as RGB, Optical Flow, Depth, and 3D positions of key joints. The promising performance of our approach is obtained through experiments conducted on three public datasets—Chalearn LAP ConGD dataset, 20BN-Jester, and NVIDIA Dynamic Hand gesture Dataset. Our approach outperforms the state-of-the-art methods on the Chalearn LAP ConGD dataset.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Human Pose and Action Recognition · Gait Recognition and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence Convergence, Chonnam National University, 77 Yongbong-ro, Gwangju 500-757, Korea
Funding
the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) · the Bio & Medical Technology Development Program of the National Research Foundation (NRF)& funded by the Korean government (MSIT)
References (37)
Show source-provided references
1. Sign Language Spotting with a Threshold Model Based on Conditional Random Fields10.1109/TPAMI.2008.172 ↗
2. Detection of gestures without begin and end markers by fitting into Bézier curves with least squares method10.1016/j.patrec.2017.10.006 ↗
3. Wang, H., Wang, P., Song, Z., and Li, W. (2017, January 22–29). Large-scale multimodal gesture recognition using heterogeneous networks. Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy.10.1109/ICCVW.2017.370 ↗
4. Chai, X., Liu, Z., Yin, F., Liu, Z., and Chen, X. (2016, January 4–8). two streams recurrent neural networks for large-scale continuous gesture recognition. Proceedings of the 2016 23rd International Conference on Pattern Recognition (ICPR), Cancun, Mexico.10.1109/ICPR.2016.7899603 ↗
5. Molchanov, P., Yang, X., Gupta, S., Kim, K., Tyree, S., and Kautz, J. (2016, January 27–30). Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural networks. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.456 ↗
6. Naguri, C.R., and Bunescu, R.C. (2017, January 18–21). Recognition of Dynamic Hand Gestures From 3D Motion Data Using LSTM and CNN Architectures. Proceedings of the 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), Cancun, Mexico.10.1109/ICMLA.2017.00013 ↗
7. Deep Dynamic Neural Networks for Multimodal Gesture Segmentation and Recognition10.1109/TPAMI.2016.2537340 ↗
8. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.510 ↗
9. Kopuklu, O., Kose, N., and Rigoll, G. (2018, January 18–22). Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA.10.1109/CVPRW.2018.00284 ↗
10. Narayana, P., Beveridge, J.R., and Draper, B.A. (2018, January 18–22). Gesture Recognition: Focus on the Hands. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00549 ↗
11. Zhu, G., Zhang, L., Mei, L., Shao, J., Song, J., and Shen, P. (2016, January 4–8). Large-scale Isolated Gesture Recognition Using Pyramidal 3D Convolutional Networks. Proceedings of the 2016 23rd International Conference on Pattern Recognition (ICPR), Cancun, Mexico.10.1109/ICPR.2016.7899601 ↗
12. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. (2006, January 25–29). Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.10.1145/1143844.1143891 ↗
13. Hara, K., Kataoka, H., and Satoh, Y. (2018, January 18–22). Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00685 ↗
14. Hoang, N.N., Lee, G.-S., Kim, S.-H., and Yang, H.-J. (2019, January 22–25). Continuous Hand Gesture Spotting and Classification Using 3D Finger Joints Information. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8803813 ↗
15. Krishnan, N.C., Lade, P., and Panchanathan, S. (2010, January 19–23). Activity gesture spotting using a threshold model based on Adaptive Boosting. Proceedings of the 2010 IEEE International Conference on Multimedia and Expo, Singapore.10.1109/ICME.2010.5583013 ↗
16. Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features10.1109/ACCESS.2017.2778011 ↗
17. Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. (2015, January 7–12). Long-term Recurrent Convolutional Networks for Visual Recognition and Description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298878 ↗
18. Du, Y., Wang, W., and Wang, L. (2015, January 7–12). Hierarchical recurrent neural network for skeleton based action recognition. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
19. Progressive Filtering Approach for Early Human Action Recognition10.1007/s12555-017-0532-1 ↗
20. Similar Hand Gesture Recognition by Automatically Extracting Distinctive Features10.1007/s12555-015-0403-6 ↗
21. A Sociable Human-robot Interaction Scheme Based on Body Emotion Analysis10.1007/s12555-017-0423-5 ↗
22. Tran, D.-S., Ho, N.-H., Yang, H.-J., Baek, E.-T., Kim, S.-H., and Lee, G. (2020). Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network. Appl. Sci., 10.10.3390/app10020722 ↗
23. Zhou, X., Huang, Q., Sun, X., Xue, X., and Wei, Y. (2017, January 22–29). Towards 3D Human Pose Estimation in The Wild: A Weakly-Supervised Approach. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.51 ↗
24. Human3. 6m: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments10.1109/TPAMI.2013.248 ↗
25. Newell, A., Yang, K., and Deng, J. (2016, January 8–16). Stacked Hourglass Networks for Human Pose Estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_29 ↗
26. Müeller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., and Theobalt, C. (2017, January 22–29). Real-Time Hand Tracking Under Occlusion from an Egocentric RGB-D Sensor. Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy.10.1109/ICCVW.2017.82 ↗
27. Zhang, J., Jiao, J., Chen, M., Qu, L., Xu, X., and Yang, Q. (2016). 3D Hand Pose Tracking and Estimation Using Stereo Matching. arXiv.
28. Farnebäck, G. (July, January 29). Two-Frame Motion Estimation Based on Polynomial Expansion. Proceedings of the Scandinavian Conference on Image Analysis, Halmstad, Sweden.
29. Kantorov, V., and Laptev, I. (2014, January 23–28). Efficient Feature Extraction, Encoding and Classification for Action Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.332 ↗
30. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. (2004, January 11–14). High Accuracy Optical Flow Estimation Based on a Theory for Warping. Proceedings of the European Conference on Computer Vision, Prague, Czech Republic.10.1007/978-3-540-24673-2_3 ↗
31. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
32. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014, January 23–28). Large-Scale Video Classification with Convolutional Neural Networks. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.223 ↗
33. Sarkar, A., Gepperth, A., Handmann, U., and Kopinski, T. (2017, January 11–13). Dynamic Hand Gesture Recognition for Mobile Systems Using Deep LSTM. Proceedings of the 9th International Conference on Intelligent Human Computer Interaction, Evry, France.10.1007/978-3-319-72038-8_3 ↗
34. Materzynska, J., Berger, G., Bax, I., and Memisevic, R. (2019, January 28–29). The Jester Dataset: A Large-Scale Video Dataset of Human Gestures. Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Korea.10.1109/ICCVW.2019.00349 ↗
35. Wan, J., Zhao, Y., Zhou, S., Guyon, I., Escalera, S., and Li, S.Z. (July, January 26). ChaLearn Looking at People RGB-D Isolated and Continuous Datasets for Gesture Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Las Vegas, NV, USA.
36. Kingma, D.P., and Ba, J. (2015, January 5–8). Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference for Learning Representation (ICLR), San Diego, CA, USA.
37. Lea, C., Vidal, R., and Hager, G.D. (2016, January 16–21). Learning convolutional action primitives for fine-grained action recognition. Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden.10.1109/ICRA.2016.7487305 ↗