Multi-Term Attention Networks for Skeleton-Based Action Recognition
Source abstract
Abstract
The same action takes different time in different cases. This difference will affect the accuracy of action recognition to a certain extent. We propose an end-to-end deep neural network called “Multi-Term Attention Networks” (MTANs), which solves the above problem by extracting temporal features with different time scales. The network consists of a Multi-Term Attention Recurrent Neural Network (MTA-RNN) and a Spatio-Temporal Convolutional Neural Network (ST-CNN). In MTA-RNN, a method for fusing multi-term temporal features are proposed to extract the temporal dependence of different time scales, and the weighted fusion temporal feature is recalibrated by the attention mechanism. Ablation research proves that this network has powerful spatio-temporal dynamic modeling capabilities for actions with different time scales. We perform extensive experiments on four challenging benchmark datasets, including the NTU RGB+D dataset, UT-Kinect dataset, Northwestern-UCLA dataset, and UWA3DII dataset. Our method achieves better results than the state-of-the-art benchmarks, which demonstrates the effectiveness of MTANs.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Anomaly Detection Techniques and Applications · Gait Recognition and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Engineering and Science, Shanghai University, Shanghai 200444, China
Funding
Science and Technology Committee of Shanghai Municipality
References (45)
Show source-provided references
1. A survey on vision-based human action recognition10.1016/j.imavis.2009.11.014 ↗
2. Yu, G., Liu, Z., and Yuan, J. (2014). Discriminative Orderlet Mining for Real-Time Recognition of Human-Object Interaction, Springer.10.1007/978-3-319-16814-2_4 ↗
3. Smisek, J., Jancosek, M., and Pajdla, T. (2013). 3D with Kinect. Consumer Depth Cameras for Computer Vision, Springer.10.1007/978-1-4471-4640-7_1 ↗
4. Coupled hidden conditional random fields for RGB-D human action recognition10.1016/j.sigpro.2014.08.038 ↗
5. Du, Y., Wang, W., and Wang, L. (2015, January 7–12). Hierarchical recurrent neural network for skeleton based action recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA.
6. Ke, Q., Bennamoun, M., An, S., Sohel, F., and Boussaid, F. (2017, January 21–26). A new representation of skeleton sequences for 3d action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.486 ↗
7. Beyond frame-level CNN: Saliency-aware 3-D CNN with LSTM for video action recognition10.1109/LSP.2016.2611485 ↗
8. Lee, I., Kim, D., Kang, S., and Lee, S. (2017, January 22–29). Ensemble deep learning for skeleton-based action recognition using temporal sliding lstm networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.115 ↗
9. Xie, C., Li, C., Zhang, B., Chen, C., Han, J., Zou, C., and Liu, J. (2018, January 13–19). Memory Attention Networks for Skeleton-based Action Recognition. Proceedings of the International Joint Conference on Artificial Intelligence 2018, Stockholm, Sweden.10.24963/ijcai.2018/227 ↗
10. Saaty, T.L. (1988). What is the analytic hierarchy process?. Mathematical Models for Decision Support, Springer.10.13033/isahp.y1988.042 ↗
11. Cao, Z., Simon, T., Wei, S.E., and Sheikh, Y. (2017, January 21–26). Realtime multi-person 2d pose estimation using part affinity fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
12. Li, W., Wen, L., Chang, M.C., Nam Lim, S., and Lyu, S. (2017, January 22–29). Adaptive RNN tree for large-scale human action recognition. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.161 ↗
13. Zhu, W., Lan, C., Xing, J., Zeng, W., Li, Y., Shen, L., and Xie, X. (2016, January 12–17). Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10451 ↗
14. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.10.3115/v1/D14-1179 ↗
15. Liu, J., Shahroudy, A., Xu, D., and Wang, G. (2016). Spatio-Temporal LSTM with Trust Gates for 3d Human Action Recognition, Springer.10.1007/978-3-319-46487-9_50 ↗
16. Luong, M.T., Pham, H., and Manning, C.D. (2015). Effective approaches to attention-based neural machine translation. arXiv.10.18653/v1/D15-1166 ↗
17. Song, S., Lan, C., Xing, J., Zeng, W., and Liu, J. (2017, January 4–9). An end-to-end spatio-temporal attention model for human action recognition from skeleton data. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11212 ↗
18. Tang, Y., Tian, Y., Lu, J., Li, P., and Zhou, J. (2018, January 18–22). Deep progressive reinforcement learning for skeleton-based action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00558 ↗
19. Hussein, M.E., Torki, M., Gowayyed, M.A., and El-Saban, M. (2013, January 3–9). Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations. Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, Beijing, China.
20. Wang, J., Liu, Z., Wu, Y., and Yuan, J. (2012, January 16–21). Mining actionlet ensemble for action recognition with depth cameras. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.
21. Vemulapalli, R., Arrate, F., and Chellappa, R. (2014, January 24–27). Human action recognition by representing 3d skeletons as points in a lie group. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2014, Columbus, OH, USA.10.1109/CVPR.2014.82 ↗
22. Li, C., Zhong, Q., Xie, D., and Pu, S. (2017, January 10–14). Skeleton-based action recognition with convolutional neural networks. Proceedings of the 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Hong Kong, China.
23. Shahroudy, A., Liu, J., Ng, T.T., and Wang, G. (July, January 26). Ntu rgb+ d: A large scale dataset for 3d human activity analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
24. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
25. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
26. Xia, L., Chen, C.C., and Aggarwal, J.K. (2012, January 16–21). View invariant human action recognition using histograms of 3d joints. Proceedings of the 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.10.1109/CVPRW.2012.6239233 ↗
27. Yang, X., and Tian, Y. (2014, January 24–27). Super normal vector for activity recognition using depth sequences. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2014, Columbus, OH, USA.10.1109/CVPR.2014.108 ↗
28. Histogram of oriented principal components for cross-view action recognition10.1109/TPAMI.2016.2533389 ↗
29. Wang, H., and Wang, L. (2017, January 21–26). Modeling temporal dynamics and spatial configurations of actions using two-stream recurrent neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.387 ↗
30. Li, C., Cui, Z., Zheng, W., Xu, C., and Yang, J. (2018, January 2–7). Spatio-temporal graph convolution for skeleton based action recognition. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, OR, USA.10.1609/aaai.v32i1.11776 ↗
31. Yan, S., Xiong, Y., and Lin, D. (2018, January 2–7). Spatial temporal graph convolutional networks for skeleton-based action recognition. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, OR, USA.10.1609/aaai.v32i1.12328 ↗
32. Beyond joints: Learning representations from primitive geometries for skeleton-based action recognition and detection10.1109/TIP.2018.2837386 ↗
33. Si, C., Jing, Y., Wang, W., Wang, L., and Tan, T. (2018, January 8–14). Skeleton-based action recognition with spatial reasoning and temporal stack learning. Proceedings of the European Conference on Computer Vision (ECCV) 2018, Munich, Germany.10.1007/978-3-030-01246-5_7 ↗
34. Li, C., Zhong, Q., Xie, D., and Pu, S. (2018). Co-occurrence feature learning from skeleton data for action recognition and detection with hierarchical aggregation. arXiv.10.24963/ijcai.2018/109 ↗
35. Caetano, C., Sena, J., Brémond, F., Dos Santos, J.A., and Schwartz, W.R. (2019, January 18–21). SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition. Proceedings of the 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Taipei, Taiwan.10.1109/AVSS.2019.8909840 ↗
36. Caetano, C., Brémond, F., and Schwartz, W.R. (2019, January 28–31). Skeleton Image Representation for 3D Action Recognition based on Tree Structure and Reference Joints. Proceedings of the 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Rio de Janeiro, Brazil.10.1109/SIBGRAPI.2019.00011 ↗
37. View adaptive neural networks for high performance skeleton-based human action recognition10.1109/TPAMI.2019.2896631 ↗
38. Wu, C., Wu, X.J., and Kittler, J. (2019, January 27–28). Spatial Residual Layer and Dense Connection Block Enhanced Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition. Proceedings of the IEEE International Conference on Computer Vision Workshops 2019, Seoul, Korea.10.1109/ICCVW.2019.00216 ↗
39. Wen, Y.H., Gao, L., Fu, H., Zhang, F.L., and Xia, S. (2019, January 29–31). Graph CNNs with motif and variable temporal block for skeleton-based action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.10.1609/aaai.v33i01.33018989 ↗
40. Liu, J., Wang, G., Hu, P., Duan, L.Y., and Kot, A.C. (2017, January 21–26). Global context-aware attention LSTM networks for 3D action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.391 ↗
41. Zhu, Y., Chen, W., and Guo, G. (2013, January 23–28). Fusing spatiotemporal features and joints for 3d action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops 2013, Portland, OR, USA.10.1109/CVPRW.2013.78 ↗
42. Anirudh, R., Turaga, P., Su, J., and Srivastava, A. (2015, January 7–12). Elastic functional coding of human actions: From vector-fields to latent variables. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298934 ↗
43. Zhang, S., Liu, X., and Xiao, J. (2017, January 24–31). On geometric features for skeleton-based action recognition using multilayer lstm networks. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.24 ↗
44. Learning actionlet ensemble for 3D human action recognition10.1109/TPAMI.2013.198 ↗
45. Enhanced skeleton visualization for view invariant human action recognition10.1016/j.patcog.2017.02.030 ↗