Pyramid Spatial-Temporal Graph Transformer for Skeleton-Based Action Recognition
Source abstract
Abstract
Although graph convolutional networks (GCNs) have shown their demonstrated ability in skeleton-based action recognition, both the spatial and the temporal connections rely too much on the predefined skeleton graph, which imposes a fixed prior knowledge for the aggregation of high-level semantic information via the graph-based convolution. Some previous GCN-based works introduced dynamic topology (vertex connection relationships) to capture flexible spatial correlations from different actions. Then, the local relationships from both the spatial and temporal domains can be captured by diverse GCNs. This paper introduces a more straightforward and more effective backbone to obtain the spatial-temporal correlation between skeleton joints with a local-global alternation pyramid architecture for skeleton-based action recognition, namely the pyramid spatial-temporal graph transformer (PGT). The PGT consists of four stages with similar architecture but different scales: graph embedding and transformer blocks. We introduce two kinds of transformer blocks in our work: the spatial-temporal transformer block and joint transformer block. In the former, spatial-temporal separated attention (STSA) is proposed to calculate the connection of the global nodes of the graph. Due to the spatial-temporal transformer block, self-attention can be performed on skeleton graphs with long-range temporal and large-scale spatial aggregation. The joint transformer block flattens the tokens in both the spatial and temporal domains to jointly capture the overall spatial-temporal correlations. The PGT is evaluated on three public skeleton datasets: the NTU RGBD 60, NTU RGBD 120 and NW-UCLA datasets. Better or comparable performance with the state of the art (SOTA) shows the effectiveness of our work.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Stroke Rehabilitation and Recovery · Context-Aware Activity Recognition Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Funding
Nature Natural Science Foundation of China
References (50)
Show source-provided references
1. Vemulapalli, R., Arrate, F., and Chellappa, R. (2014, January 23–28). Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA.10.1109/CVPR.2014.82 ↗
2. Enhanced skeleton visualization for view invariant human action recognition10.1016/j.patcog.2017.02.030 ↗
3. Li, B., Dai, Y., Cheng, X., Chen, H., Lin, Y., and He, M. (2017, January 10–14). Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN. Proceedings of the 2017 IEEE International Conference on Multimedia & Expo Workshops, ICME Workshops, Hong Kong, China.
4. Li, C., Zhong, Q., Xie, D., and Pu, S. (2017, January 10–14). Skeleton-based action recognition with convolutional neural networks. Proceedings of the 2017 IEEE International Conference on Multimedia & Expo Workshops, ICME Workshops, Hong Kong, China.
5. View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition10.1109/TPAMI.2019.2896631 ↗
6. Liu, J., Shahroudy, A., Xu, D., and Wang, G. (2016, January 11–14). Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition. Proceedings of the Computer Vision—ECCV 2016—14th European Conference, Amsterdam, The Netherlands. Lecture Notes in Computer Science.10.1007/978-3-319-46487-9_50 ↗
7. Song, S., Lan, C., Xing, J., Zeng, W., and Liu, J. (2017, January 4–9). An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11212 ↗
8. Zhang, P., Lan, C., Xing, J., Zeng, W., Xue, J., and Zheng, N. (2017, January 22–29). View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy.10.1109/ICCV.2017.233 ↗
9. Zheng, W., Li, L., Zhang, Z., Huang, Y., and Wang, L. (2019, January 8–12). Relational Network for Skeleton-Based Action Recognition. Proceedings of the IEEE International Conference on Multimedia and Expo, ICME 2019, Shanghai, China.10.1109/ICME.2019.00147 ↗
10. Yan, S., Xiong, Y., and Lin, D. (2018, January 2–7). Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12328 ↗
11. Shi, L., Zhang, Y., Cheng, J., and Lu, H. (2019, January 16–20). Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA.10.1109/CVPR.2019.01230 ↗
12. Cheng, K., Zhang, Y., He, X., Chen, W., Cheng, J., and Lu, H. (2020, January 13–19). Skeleton-Based Action Recognition With Shift Graph Convolutional Network. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA.10.1109/CVPR42600.2020.00026 ↗
13. Richly Activated Graph Convolutional Network for Robust Skeleton-Based Action Recognition10.1109/TCSVT.2020.3015051 ↗
14. Skeleton-based action recognition via spatial and temporal transformer networks10.1016/j.cviu.2021.103219 ↗
15. Song, Y.F., Zhang, Z., Shan, C., and Wang, L. (2022). Constructing stronger and faster baselines for skeleton-based action recognition. IEEE Trans. Pattern Anal. Mach. Intell., online ahead of print.10.1109/TPAMI.2022.3157033 ↗
16. Hussein, M.E., Torki, M., Gowayyed, M.A., and El-Saban, M. (2013, January 3–9). Human Action Recognition Using a Temporal Hierarchy of Covariance Descriptors on 3D Joint Locations. Proceedings of the 23rd International Joint Conference on Artificial Intelligence, IJCAI 2013, Beijing, China.
17. Kim, T.S., and Reiter, A. (2017, January 21–26). Interpretable 3D Human Action Analysis with Temporal Convolutional Networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA.10.1109/CVPRW.2017.207 ↗
18. Fuzzy Integral-Based CNN Classifier Fusion for 3D Skeleton Action Recognition10.1109/TCSVT.2020.3019293 ↗
19. Skeleton edge motion networks for human action recognition10.1016/j.neucom.2020.10.037 ↗
20. Diao, X., Li, X., and Huang, C. (2020). Multi-term attention networks for skeleton-based action recognition. Appl. Sci., 10.10.3390/app10155326 ↗
21. Action Recognition Scheme Based on Skeleton Representation With DS-LSTM Network10.1109/TCSVT.2019.2914137 ↗
22. Gao, Y., Li, C., Li, S., Cai, X., Ye, M., and Yuan, H. (2022). A Deep Attention Model for Action Recognition from Skeleton Data. Appl. Sci., 12.10.3390/app12042006 ↗
23. Liu, Z., Zhang, H., Chen, Z., Wang, Z., and Ouyang, W. (2020, January 13–19). Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA.10.1109/CVPR42600.2020.00022 ↗
24. Huang, Z., Shen, X., Tian, X., Li, H., Huang, J., and Hua, X. (2020, January 12–16). Spatio-Temporal Inception Graph Convolutional Networks for Skeleton-Based Action Recognition. Proceedings of the MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event/Seattle, WA, USA.10.1145/3394171.3413666 ↗
25. Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., and Tian, Q. (2019, January 16–20). Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA.10.1109/CVPR.2019.00371 ↗
26. Dual-domain graph convolutional networks for skeleton-based action recognition10.1007/s10994-022-06141-8 ↗
27. Zheng, Z., Wang, Y., Zhang, X., and Wang, J. (2022). Multi-Scale Adaptive Aggregate Graph Convolutional Network for Skeleton-Based Action Recognition. Appl. Sci., 12.10.3390/app12031402 ↗
28. Skeleton-Based Action Recognition With Multi-Stream Adaptive Graph Convolutional Networks10.1109/TIP.2020.3028207 ↗
29. Chen, Y., Zhang, Z., Yuan, C., Li, B., Deng, Y., and Hu, W. (2021, January 10–17). Channel-wise topology refinement graph convolution for skeleton-based action recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01311 ↗
30. Cheng, K., Zhang, Y., Cao, C., Shi, L., Cheng, J., and Lu, H. (2020, January 23–28). Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition. Proceedings of the Computer Vision—ECCV 2020—16th European Conference, Glasgow, UK. Part XXIV.10.1007/978-3-030-58586-0_32 ↗
31. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA.
32. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria.
33. Srinivas, A., Lin, T., Parmar, N., Shlens, J., Abbeel, P., and Vaswani, A. (2021, January 19–25). Bottleneck Transformers for Visual Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual Event.10.1109/CVPR46437.2021.01625 ↗
34. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-End Object Detection with Transformers. Proceedings of the Computer Vision—ECCV 2020—16th European Conference, Glasgow, UK. Part I.10.1007/978-3-030-58452-8_13 ↗
35. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2021, January 18–24). Training data-efficient image transformers & distillation through attention. Proceedings of the 38th International Conference on Machine Learning, ICML 2021, Virtual Event. Proceedings of Machine Learning Research.
36. Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z.H., Tay, F.E., Feng, J., and Yan, S. (2021, January 10–17). Tokens-to-token vit: Training vision transformers from scratch on imagenet. Proceedings of the IEEE/CVF International Conference on Computer Vision 2021, Montreal, QC, Canada.10.1109/ICCV48922.2021.00060 ↗
37. Wang, W., Xie, E., Li, X., Fan, D., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. (2021, January 10–17). Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada.10.1109/ICCV48922.2021.00061 ↗
38. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
39. Zhang, Y., Wu, B., Li, W., Duan, L., and Gan, C. (2022, January 10–14). STST: Spatial-Temporal Specialized Transformer for Skeleton-based Action Recognition. Proceedings of the 29th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3474085.3475473 ↗
40. Shahroudy, A., Liu, J., Ng, T.T., and Wang, G. (2016, January 27–30). NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA.10.1109/CVPR.2016.115 ↗
41. NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding10.1109/TPAMI.2019.2916873 ↗
42. Wang, J., Nie, X., Xia, Y., Wu, Y., and Zhu, S. (2014, January 23–28). Cross-View Action Modeling, Learning, and Recognition. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA.10.1109/CVPR.2014.339 ↗
43. Zhang, P., Lan, C., Zeng, W., Xing, J., Xue, J., and Zheng, N. (2020, January 13–19). Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA.10.1109/CVPR42600.2020.00119 ↗
44. Cho, S., Maqbool, M.H., Liu, F., and Foroosh, H. (2020, January 1–5). Self-Attention Network for Skeleton-based Human Action Recognition. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA.10.1109/WACV45572.2020.9093639 ↗
45. Adaptive multi-view graph convolutional networks for skeleton-based action recognition10.1016/j.neucom.2020.03.126 ↗
46. Caetano, C., de Souza, J.S., Brémond, F., dos Santos, J.A., and Schwartz, W.R. (2019, January 18–21). SkeleMotion: A New Representation of Skeleton Joint Sequences based on Motion Information for 3D Action Recognition. Proceedings of the 16th IEEE International Conference on Advanced Video and Signal Based Surveillance, AVSS 2019, Taipei, Taiwan.10.1109/AVSS.2019.8909840 ↗
47. Caetano, C., Brémond, F., and Schwartz, W.R. (2019, January 28–30). Skeleton image representation for 3d action recognition based on tree structure and reference joints. Proceedings of the 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Rio de Janeiro, Brazil.10.1109/SIBGRAPI.2019.00011 ↗
48. Das, S., Dai, R., Koperski, M., Minciullo, L., Garattoni, L., Bremond, F., and Francesca, G. (November, January 27). Toyota smarthome: Real-world activities of daily living. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
49. Das, S., Sharma, S., Dai, R., Bremond, F., and Thonnat, M. (2020). VPN: Learning video-pose embedding for activities of daily living. European Conference on Computer Vision, Springer.10.1007/978-3-030-58545-7_5 ↗
50. Lee, I., Kim, D., Kang, S., and Lee, S. (2017, January 22–29). Ensemble deep learning for skeleton-based action recognition using temporal sliding lstm networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.115 ↗