Human Action Recognition Based on 3D Convolution and Multi-Attention Transformer
Source abstract
Abstract
To address the limitations of traditional two-stream networks, such as inadequate spatiotemporal information fusion, limited feature diversity, and insufficient accuracy, we propose an improved two-stream network for human action recognition based on multi-scale attention Transformer and 3D convolutional (C3D) fusion. In the temporal stream, the traditional 2D convolutional is replaced with a C3D network to effectively capture temporal dynamics and spatial features. In the spatial stream, a multi-scale convolutional Transformer encoder is introduced to extract features. Leveraging the multi-scale attention mechanism, the model captures and enhances features at various scales, which are then adaptively fused using a weighted strategy to improve feature representation. Furthermore, through extensive experiments on feature fusion methods, the optimal fusion strategy for the two-stream network is identified. Experimental results on benchmark datasets such as UCF101 and HMDB51 demonstrate that the proposed model achieves superior performance in action recognition tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Gait Recognition and Analysis · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Information Science & Technology, Qingdao University of Science & Technology, Qingdao 266101, China
College of Information Engineering, Xinjiang Institute of Engineering, Urumqi 830023, China
Funding
National Natural Science Foundation · Key Research and Development Foundation of Shandong Province
References (40)
Show source-provided references
1. Duan, H., Zhao, Y., Chen, K., Lin, D., and Dai, B. (2022, January 18–24). Revisiting skeleton-based action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00298 ↗
2. 3D Deformable Convolution Temporal Reasoning network for action recognition10.1016/j.jvcir.2023.103804 ↗
3. Research on 3D convolutional neural network and its application to video understanding
4. Skeleton-based action recognition through contrasting two-stream spatial temporal networks10.1109/TMM.2023.3239751 ↗
5. Mask guided two-stream network for end-to-end few-shot action recognition10.1016/j.neucom.2024.127582 ↗
6. STDM-transformer: Spacetime dual multi-scale transformer network for skeleton based action recognition10.1016/j.neucom.2023.126903 ↗
7. Transformer-based multiview spatiotemporal feature interactive fusion for human action recognition in depth videos10.1016/j.image.2024.117244 ↗
8. Xie, J., Meng, Y., Zhao, Y., Nguyen, A., Yang, X., and Zheng, Y. (2024, January 20–27). Dynamic Semantic-Based Spatial Graph Convolution Network for Skeleton-Based Human Action Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i6.28440 ↗
9. Liu, H., Liu, Y., Ren, M., Wang, H., Wang, Y., and Sun, Z. (2024). Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition. arXiv.
10. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 13–16). Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.510 ↗
11. Qiu, Z., Yao, T., and Mei, T. (2017, January 22–29). Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.590 ↗
12. Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., and Paluri, M. (2018, January 18–23). A closer look at spatiotemporal convolutions for action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00675 ↗
13. Yang, F., Li, S., Sun, C., Li, X., and Xiao, Z. (2024). Action recognition in rehabilitation: Combining 3D convolution and LSTM with spatiotemporal attention. Front. Physiol., 15.10.3389/fphys.2024.1472380 ↗
14. Feichtenhofer, C., Fan, H., Malik, J., and He, K. (November, January 27). SlowFast Networks for Video Recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV), Seoul, Republic of Korea.
15. Rethinking Motion Representation: Residual Frames With 3D ConvNets10.1109/TIP.2021.3124156 ↗
16. FSformer: Fast-Slow Transformer for video action recognition10.1016/j.imavis.2023.104740 ↗
17. Multi-head attention-based two-stream EfficientNet for action recognition10.1007/s00530-022-00961-3 ↗
18. Modeling transformer architecture with attention layer for human activity recognition10.1007/s00521-023-09362-7 ↗
19. Relative-position embedding based spatially and temporally decoupled Transformer for action recognition10.1016/j.patcog.2023.109905 ↗
20. Qiu, H., Hou, B., Ren, B., and Zhang, X. (2022). Spatio-temporal tuples transformer for skeleton-based action recognition. arXiv.10.1016/j.neucom.2022.10.084 ↗
21. Do, J., and Kim, M. (October, January 9). SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy.
22. TranSkeleton: Hierarchical Spatial–Temporal Transformer for Skeleton-Based Action Recognition10.1109/TCSVT.2023.3240472 ↗
23. k-NN attention-based video vision transformer for action recognition10.1016/j.neucom.2024.127256 ↗
24. Zhang, C., Wu, J., and Li, Y. (2022, January 23–27). ActionFormer: Localizing Moments of Actions with Transformers. Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel.10.1007/978-3-031-19772-7_29 ↗
25. Zheng, Y., Li, Z., Wang, Z., and Wu, L. (2023, January 4–6). Transformer-Based Two-Stream Network for Global and Local Motion Estimation. Proceedings of the IEEE International Conference on Pattern Recognition and Machine Learning (PRML), Urumqi, China.10.1109/PRML59573.2023.10348241 ↗
26. Yang, J., Dong, X., Liu, L., Zhang, C., Shen, J., and Yu, D. (2022, January 18–24). Recurring the Transformer for Video Action Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01367 ↗
27. Neimark, D., Bar, O., Zohar, M., and Asselmann, D. (2021, January 11–17). Video Transformer Network. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Virtual.10.1109/ICCVW54120.2021.00355 ↗
28. Multi-Scale Adaptive Skeleton Transformer for action recognition10.1016/j.cviu.2024.104229 ↗
29. MTT: Multi-Scale Temporal Transformer for Skeleton-Based Action Recognition10.1109/LSP.2022.3142675 ↗
30. Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv.
31. Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T. (2011, January 1–8). HMDB: A large video database for human motion recognition. Proceedings of the IEEE Conference on Computer Vision(ICCV), Sydney, Australia.10.1109/ICCV.2011.6126543 ↗
32. Schuldt, C., Laptev, I., and Caputo, B. (2004, January 23–26). Recognizing Human Actions: A Local SVM Approach. Proceedings of the 17th International Conference on Pattern Recognition (ICPR), Cambridge, UK.10.1109/ICPR.2004.1334462 ↗
33. Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., and Natsev, P. (2017). The kinetics human action video dataset. arXiv.
34. Feichtenhofer, C., Pinz, A., and Zisserman, A. (2016, January 27–30). Convolutional Two-Stream Network Fusion for Video Action Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.213 ↗
35. Wang, H., and Schmid, C. (2013, January 1–8). Action recognition with improved trajectories. Proceedings of the IEEE Conference on Computer Vision (ICCV), Sydney, Australia.10.1109/ICCV.2013.441 ↗
36. Morgado, P., Vasconcelos, N., and Misra, I. (2021, January 18–24). Audio-visual Instance Discrimination with Cross-Modal-Agreement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR46437.2021.01274 ↗
37. Wang, L., Tong, Z., Ji, B., and Wu, G. (2021, January 20–25). TDN: Temporal Difference Networks for Efficient Action Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00193 ↗
38. Spatiotemporal saliency-based multi-stream networks with attention-aware LSTM for action recognition10.1007/s00521-020-05144-7 ↗
39. Manh, N.D.D., Hang, D.V., and Wang, J.C. (2024). YOWOv3: An Efficient and Generalized Framework for Human Action Detection and Recognition. arXiv.
40. Cascade multi-head attention networks for action recognition10.1016/j.cviu.2019.102898 ↗