A Deep Attention Model for Action Recognition from Skeleton Data
Source abstract
Abstract
This paper presents a new IndRNN-based deep attention model, termed DA-IndRNN, for skeleton-based action recognition to effectively model the fact that different joints are usually of different degrees of importance to different action categories. The model consists of (a) a deep IndRNN as the main classification network to overcome the limitation of a shallow RNN network in order to obtain deeper and longer features, and (b) a deep attention network with multiple fully connected layers to estimate reliable attention weights. To train the DA-IndRNN, a new triplet loss function is proposed to guide the learning of the attention among different action categories. Specifically, this triplet loss enforces intra-class attention distances to be smaller than inter-class attention distances and at the same time to allow multiple attention weight patterns to exist for the same class. The proposed DA-IndRNN can be trained end-to-end. Experiments on the widely used datasets, including the NTU RGB + D dataset and UOW Large-Scale Combined (LSC) Dataset, have demonstrated that the proposed method can achieve better and stable performance than the state-of-the-art attention models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Gait Recognition and Analysis · Diabetic Foot Ulcer Assessment and Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Software, Shandong University, Jinan 250101, China
Shandong University-Weihai Research Institute of Industrial Technology, Weihai 264209, China
State Key Laboratory of Dynamic Testing Technology, School of Information and Communication Engineering, North University of China, Taiyuan 030051, China
School of Control Science and Engineering, Shandong University, Jinan 250100, China
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Funding
National Natural Science Foundation of China · Fundamental Research Program of Shanxi Province · National Key R&D Program of China · open project program of state key laboratory of virtual reality technology and systems, Beihang University
References (57)
Show source-provided references
1. 3D convolutional neural networks for human action recognition10.1109/TPAMI.2012.59 ↗
2. Wang, P., Li, W., Gao, Z., Zhang, Y., Tang, C., and Ogunbona, P. (2017, January 21–26). Scene flow to action map: A new representation for RGB-D based action recognition with convolutional neural networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.52 ↗
3. 3-d human action recognition by shape analysis of motion trajectories on riemannian manifold10.1109/TCYB.2014.2350774 ↗
4. Enhanced skeleton visualization for view invariant human action recognition10.1016/j.patcog.2017.02.030 ↗
5. Semi-supervised image-to-video adaptation for video action recognition10.1109/TCYB.2016.2535122 ↗
6. Du, W., Wang, Y., and Qiao, Y. (2017, January 22–29). Rpan: An end-to-end recurrent pose-attention network for action recognition in videos. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.402 ↗
7. Girdhar, R., and Ramanan, D. (2017, January 4–9). Attentional pooling for action recognition. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
8. RGB-D-based human motion recognition with deep learning: A survey10.1016/j.cviu.2018.04.007 ↗
9. Depth Pooling Based Large-Scale 3-D Action Recognition with Convolutional Neural Networks10.1109/TMM.2018.2818329 ↗
10. Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. (2015, January 7–12). Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298878 ↗
11. Yue-Hei Ng, J., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., and Toderici, G. (2015, January 7–12). Beyond short snippets: Deep networks for video classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299101 ↗
12. Beyond temporal pooling: Recurrence and temporal convolutions for gesture recognition in video10.1007/s11263-016-0957-7 ↗
13. Jain, A., Zamir, A.R., Savarese, S., and Saxena, A. (2016, January 27–30). Structural-RNN: Deep learning on spatio-temporal graphs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.573 ↗
14. Li, S., Li, W., Cook, C., Zhu, C., and Gao, Y. (2018, January 18–2). Independently recurrent neural network (indrnn): Building A longer and deeper RNN. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00572 ↗
15. Li, C., Li, S., Gao, Y., Zhang, X., and Li, W. (2021). A Two-stream Neural Network for Pose-based Hand Gesture Recognition. IEEE Trans. Cogn. Dev. Syst.10.1109/TCDS.2021.3126637 ↗
16. Zhao, B., Li, S., Gao, Y., Li, C., and Li, W. (2020). A Framework of Combining Short-Term Spatial/Frequency Feature Extraction and Long-Term IndRNN for Activity Recognition. Sensors, 20.10.3390/s20236984 ↗
17. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015, January 6–11). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the International Conference on Machine Learning, Lille, France.
18. Sharma, S., Kiros, R., and Salakhutdinov, R. (2015). Action recognition using visual attention. arXiv.
19. Song, S., Lan, C., Xing, J., Zeng, W., and Liu, J. (2017, January 4–9). An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11212 ↗
20. Liu, J., Wang, G., Hu, P., Duan, L.Y., and Kot, A.C. (2017, January 21–26). Global context-aware attention lstm networks for 3D action recognition. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.391 ↗
21. Pascanu, R., Mikolov, T., and Bengio, Y. (2013, January 16–21). On the difficulty of training recurrent neural networks. Proceedings of the International Conference on Machine Learning (ICML), Atlanta, GA, USA.
22. Wang, P., Li, Z., Hou, Y., and Li, W. (2016, January 15–19). Action recognition based on joint trajectory maps using convolutional neural networks. Proceedings of the 2016 ACM on Multimedia Conference, Amsterdam, The Netherlands.10.1145/2964284.2967191 ↗
23. SkeletonNet: Mining Deep Part Features for 3-D Action Recognition10.1109/LSP.2017.2690339 ↗
24. Kim, T.S., and Reiter, A. (2017). Interpretable 3D Human Action Analysis with Temporal Convolutional Networks. arXiv.10.1109/CVPRW.2017.207 ↗
25. Ke, Q., Bennamoun, M., An, S., Sohel, F., and Boussaid, F. (2017). A New Representation of Skeleton Sequences for 3D Action Recognition. arXiv.10.1109/CVPR.2017.486 ↗
26. Body Joint Guided 3-D Deep Convolutional Descriptors for Action Recognition10.1109/TCYB.2017.2756840 ↗
27. Du, Y., Wang, W., and Wang, L. (2015, January 7–12). Hierarchical recurrent neural network for skeleton based action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
28. Veeriah, V., Zhuang, N., and Qi, G.J. (2015, January 7–13). Differential recurrent neural networks for action recognition. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.460 ↗
29. Shahroudy, A., Liu, J., Ng, T.T., and Wang, G. (2016, January 27–30). NTU RGB + D: A large scale dataset for 3D human activity analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.115 ↗
30. Liu, J., Shahroudy, A., Xu, D., and Wang, G. (2016, January 11–14). Spatio-temporal LSTM with trust gates for 3D human action recognition. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46487-9_50 ↗
31. Zhang, S., Liu, X., and Xiao, J. (2017, January 24–31). On geometric features for skeleton-based action recognition using multilayer LSTM networks. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.24 ↗
32. Joint Distance Maps Based Action Recognition with Convolutional Neural Networks10.1109/LSP.2017.2678539 ↗
33. Skeleton Optical Spectra-Based Action Recognition Using Convolutional Neural Networks10.1109/TCSVT.2016.2628339 ↗
34. Liu, J.W., Akhtar, N., and Mian, A.S. (2019). Skepxels: Spatio-temporal Image Representation of Human Skeleton Joints for Action Recognition. arXiv.
35. 3D skeleton based action recognition by video-domain translation-scale invariant mapping and multi-scale dilated CNN10.1007/s11042-018-5642-0 ↗
36. Zhu, W., Lan, C., Xing, J., Zeng, W., Li, Y., Shen, L., and Xie, X. (2016, January 12–17). Co-Occurrence Feature Learning for Skeleton Based Action Recognition Using Regularized Deep LSTM Networks. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10451 ↗
37. Ba, J., Mnih, V., and Kavukcuoglu, K. (2014). Multiple object recognition with visual attention. arXiv.
38. Haque, A., Alahi, A., and Li, F.-F. (2016, January 27–30). Recurrent attention models for depth-based person identification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.138 ↗
39. Li, S., Li, W., Cook, C., and Gao, Y. (2019). Deep Independently Recurrent Neural Network (IndRNN). arXiv.10.1109/CVPR.2018.00572 ↗
40. Liu, Z., Zhang, H., Chen, Z., Wang, Z., and Ouyang, W. (2020, January 14–19). Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00022 ↗
41. Chen, Y., Zhang, Z., Yuan, C., Li, B., Deng, Y., and Hu, W. (2021). Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition. arXiv.10.1109/ICCV48922.2021.01311 ↗
42. Schroff, F., Kalenichenko, D., and Philbin, J. (2015, January 7–12). Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298682 ↗
43. Hoffer, E., and Ailon, N. (2015, January 28–30). Deep metric learning using triplet network. Proceedings of the International Workshop on Similarity-Based Pattern Recognition, Venice, Italy.10.1007/978-3-319-24261-3_7 ↗
44. Zhang, J., Li, W., Wang, P., Ogunbona, P., Liu, S., and Tang, C. (2016, January 4). A large scale rgb-d dataset for action recognition. Proceedings of the International Workshop on Understanding Human Activities through 3D Sensors, Cancun, Mexico.
45. Huang, Z., Wan, C., Probst, T., and Van Gool, L. (2016). Deep learning on lie groups for skeleton-based action recognition. arXiv.10.1109/CVPR.2017.137 ↗
46. Yan, S., Xiong, Y., and Lin, D. (2018). Spatial temporal graph convolutional networks for skeleton-based action recognition. arXiv.10.1609/aaai.v32i1.12328 ↗
47. Li, W., Zhang, Z., and Liu, Z. (2010, January 13–18). Action recognition based on a bag of 3D points. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543273 ↗
48. Xia, L., Chen, C.C., and Aggarwal, J.K. (2012, January 16–21). View invariant human action recognition using histograms of 3D joints. Proceedings of the 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.10.1109/CVPRW.2012.6239233 ↗
49. Wang, J., Liu, Z., Wu, Y., and Yuan, J. (2012, January 16–21). Mining actionlet ensemble for action recognition with depth cameras. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.
50. Oreifej, O., and Liu, Z. (2013, January 23–28). Hon4d: Histogram of oriented 4d normals for activity recognition from depth sequences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.98 ↗
51. Yang, X., and Tian, Y. (2014, January 23–28). Super normal vector for activity recognition using depth sequences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.108 ↗
52. Sung, J., Ponce, C., Selman, B., and Saxena, A. (2011). Human Activity Detection from RGBD Images. Plan, Activity, and Intent Recognition, ACM Digital Library.
53. Bloom, V., Makris, D., and Argyriou, V. (2012, January 16–21). G3D: A gaming action dataset and real time action recognition evaluation framework. Proceedings of the 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.10.1109/CVPRW.2012.6239175 ↗
54. Ni, B., Wang, G., and Moulin, P. (2011, January 6–13). Rgbd-hudaact: A color-depth video database for human daily activity recognition. Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.10.1109/ICCVW.2011.6130379 ↗
55. Chen, C., Jafari, R., and Kehtarnavaz, N. (2015, January 27–30). Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada.10.1109/ICIP.2015.7350781 ↗
56. Ohn-Bar, E., and Trivedi, M. (2013, January 23–28). Joint angles similarities and HOG2 for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Portland, OR, USA.10.1109/CVPRW.2013.76 ↗
57. Action-Attending Graphic Neural Network10.1109/TIP.2018.2815744 ↗