Multi-Grained Temporal Clip Transformer for Skeleton-Based Human Activity Recognition
Source abstract
Abstract
Skeleton-based human activity recognition is a key research topic in the fields of deep learning and computer vision. However, existing approaches are less effective at capturing short-term sub-action information at different granularity levels and long-term motion correlations, which affect recognition accuracy. To overcome these challenges, an innovative multi-grained temporal clip transformer (MTC-Former) is proposed. Firstly, based on the transformer backbone, a multi-grained temporal clip attention (MTCA) module with multi-branch architecture is proposed to capture the characteristics of short-term sub-action features. Secondly, an innovative multi-scale spatial–temporal feature interaction module is proposed to jointly learn sub-action dependencies and facilitate skeletal motion interactions, where long-range motion patterns are embedded to enhance correlation modeling. Experiments were conducted on three datasets, including NTU RGB+D, NTU RGB+D 120, and InHARD, and achieved state-of-the-art Top-1 recognition accuracy, demonstrating the superiority of the proposed MTC-Former.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Gait Recognition and Analysis · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Electrical Engineering and New Energy, China Three Gorges University, Yichang 443002, China
School of Electrical and Control Engineering, Henan University of Urban Construction, Pingdingshan 467036, China
Funding
Science and Technology Development of Henan Province · National Natural Science Foundation of China · Soft Science Research Program of Henan Province
References (55)
Show source-provided references
1. An efficient dimension reduction based fusion of CNN and SVM model for detection of abnormal incident in video surveillance10.36548/jscp.2021.2.001 ↗
2. Toward proactive human—Robot collaborative assembly: A multimodal transfer-learning-enabled action prediction approach10.1109/TIE.2021.3105977 ↗
3. Simulation of artificial intelligence robots in dance action recognition and interaction process based on machine vision10.1016/j.entcom.2024.100773 ↗
4. Foot kinematics in runners with plantar heel pain during running gait10.1016/j.gaitpost.2023.05.019 ↗
5. Human action recognition from various data modalities: A review
6. RGB-D-based action recognition datasets: A survey10.1016/j.patcog.2016.05.019 ↗
7. Cao, Z., Simon, T., Wei, S., and Sheikh, Y. (2017, January 21–26). Realtime multi-person 2d pose estimation using part affinity fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
8. Shahroudy, A., Liu, J., Ng, T.T., and Wang, G. (2016, January 27–30). Ntu rgb+ d: A large scale dataset for 3d human activity analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.115 ↗
9. Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding10.1109/TPAMI.2019.2916873 ↗
10. Yan, S., Xiong, Y., and Lin, D. (2018, January 2–7). Spatial temporal graph convolutional networks for skeleton-based action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12328 ↗
11. Liu, Z., Zhang, H., Chen, Z., Wang, Z., and Ouyang, W. (2020, January 13–19). Disentangling and unifying graph convolutions for skeleton-based action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00022 ↗
12. Chen, Y., Zhang, Z., Yuan, C., Li, B., Deng, Y., and Hu, W. (2021, January 10–17). Channel-wise topology refinement graph convolution for skeleton-based action recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.01311 ↗
13. Multi-scale Structural Graph Convolutional Network for Skeleton-based Action Recognition10.1109/TCSVT.2024.3375512 ↗
14. Local and global self-attention enhanced graph convolutional network for skeleton-based action recognition10.1016/j.patcog.2024.111106 ↗
15. Lee, J., Lee, M., Lee, D., and Lee, S. (2023, January 2–6). Hierarchically decomposed graph convolutional networks for skeleton-based action recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00958 ↗
16. Skeleton-based action recognition via spatial and temporal transformer networks10.1016/j.cviu.2021.103219 ↗
17. Wang, Q., Shi, S., He, J., Peng, J., Liu, T., and Weng, R. (2023, January 15–18). Iip-transformer: Intra-inter-part transformer for skeleton-based action recognition. Proceedings of the 2023 IEEE International Conference on Big Data (BigData), Sorrento, Italy.10.1109/BigData59044.2023.10386970 ↗
18. Spatio-temporal segments attention for skeleton-based action recognition10.1016/j.neucom.2022.10.084 ↗
19. TranSkeleton: Hierarchical spatial-temporal transformer for skeleton-based action recognition10.1109/TCSVT.2023.3240472 ↗
20. STSD: Spatial–temporal semantic decomposition transformer for skeleton-based action recognition10.1007/s00530-023-01251-2 ↗
21. STDM-transformer: Space-time dual multi-scale transformer network for skeleton-based action recognition10.1016/j.neucom.2023.126903 ↗
22. Local and Global Spatial-Temporal Transformer for skeleton-based action recognition10.1016/j.neucom.2025.129820 ↗
23. Vemulapalli, R., Arrate, F., and Chellappa, R. (2014, January 23–28). Human action recognition by representing 3d skeletons as points in a lie group. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.82 ↗
24. Skeleton optical spectra-based action recognition using convolutional neural networks10.1109/TCSVT.2016.2628339 ↗
25. Zhang, P., Lan, C., Xing, J., Zeng, W., Xue, J., and Zheng, N. (2017, January 22–29). View adaptive recurrent neural networks for high performance human action recognition from skeleton data. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.233 ↗
26. Zhu, W., Lan, C., Xing, J., Zeng, W., Li, Y., Shen, L., and Xie, X. (2016, January 12–17). Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10451 ↗
27. Zhang, P., Xue, J., Lan, C., Zeng, W., Gao, Z., and Zheng, N. (2018, January 8–14). Adding attentiveness to the neurons in recurrent neural networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01240-3_9 ↗
28. Cai, D., Kang, Y., Yao, A., and Chen, Y. (2023, January 23–29). Ske2Grid: Skeleton-to-grid representation learning for action recognition. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA.
29. Temporal-channel attention and convolution fusion for skeleton-based human action recognition10.1109/ACCESS.2024.3389499 ↗
30. Li, B., Dai, Y., Cheng, X., Chen, H., Lin, Y., and He, M. (2017, January 10–14). Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN. Proceedings of the 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Hong Kong, China.
31. Duan, H., Wang, J., Chen, K., and Lin, D. (2022). Dg-stgcn: Dynamic spatial-temporal modeling for skeleton-based action recognition. arXiv.
32. Shi, L., Zhang, Y., Cheng, J., and Lu, H. (2019, January 16–20). Two-stream adaptive graph convolutional networks for skeleton-based action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01230 ↗
33. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
34. Multi-grained clip focus for skeleton-based action recognition10.1016/j.patcog.2023.110188 ↗
35. Lin, Z., Gao, Y., and Li, D. (2023). Cross-Attention Multi-Scale Spatial Temporal Transformer for Skeleton-Based Action Recognition, Springer.10.21203/rs.3.rs-3368402/v1 ↗
36. Plizzari, C., Cannici, M., and Matteucci, M. (2021, January 10–15). Spatial temporal transformer network for skeleton-based action recognition. Proceedings of the Pattern Recognition. ICPR International Workshops and Challenges, Virtual Event.10.1007/978-3-030-68796-0_50 ↗
37. Shi, L., Zhang, Y., Cheng, J., and Lu, H. (December, January 30). Decoupled spatial-temporal attention network for skeleton-based action-gesture recognition. Proceedings of the Asian Conference on Computer Vision, Kyoto, Japan.
38. Gao, Z., Wang, P., Lv, P., Jiang, X., Liu, Q., Wang, P., Xu, M., and Li, W. (2022, January 4–8). Focal and global spatial-temporal transformer for skeleton-based action recognition. Proceedings of the Asian Conference on Computer Vision, Macau, China.10.1007/978-3-031-26316-3_10 ↗
39. Irani, H., and Metsis, V. (2025). Positional Encoding in Transformer-Based Time Series Models: A Survey. arXiv.
40. Li, X., Wang, W., Hu, X., and Yang, J. (2019, January 15–20). Selective Kernel Networks. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00060 ↗
41. A review of “The Co-ordination and Regulation of Movements” By N. Bernstein. (Pergamon Press, 1967.) [Pp. xii+ 196.] 50510.1080/00140136808930945 ↗
42. Dallel, M., Havard, V., Baudry, D., and Savatier, X. (2020, January 7–9). InHARD—Industrial Human Action Recognition Dataset in the Context of Industrial Collaborative Robotics. Proceedings of the 2020 IEEE International Conference on Human-Machine Systems (ICHMS), Rome, Italy.10.1109/ICHMS49158.2020.9209531 ↗
43. Learning clip representations for skeleton-based 3D action recognition10.1109/TIP.2018.2812099 ↗
44. Xu, K., Ye, F., Zhong, Q., and Xie, D. (March, January 22). Topology-aware convolutional neural network for efficient skeleton-based action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual Event.
45. SiT-MLP: A simple MLP with point-wise topology feature learning for skeleton-based action recognition10.1109/TCSVT.2024.3386553 ↗
46. Zhou, H., Liu, Q., and Wang, Y. (2023, January 18–22). Learning discriminative representations for skeleton based action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01022 ↗
47. Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., and Tian, Q. (2019, January 16–20). Actional-structural graph convolutional networks for skeleton-based action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00371 ↗
48. Cheng, K., Zhang, Y., He, X., Chen, W., Cheng, J., and Lu, H. (2020, January 13–19). Skeleton-based action recognition with shift graph convolutional network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00026 ↗
49. Spatiotemporal progressive inward-outward aggregation network for skeleton-based action recognition10.1016/j.patcog.2024.110262 ↗
50. Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., and Van Gool, L. (2016, January 8–16). Temporal segment networks: Towards good practices for deep action recognition. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_2 ↗
51. Lin, J., Gan, C., and Han, S. (November;, January 27). Tsm: Temporal shift module for efficient video understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
52. Carreira, J., and Zisserman, A. (2017, January 21–26). Quo vadis, action recognition? a new model and the kinetics dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
53. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–22). Non-local neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
54. Bertasius, G., Wang, H., and Torresani, L. (2021, January 18–24). Is space-time attention all you need for video understanding?. Proceedings of the ICML, Vienna, Austria.
55. Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., and Hu, H. (2022, January 19–24). Video swin transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00320 ↗