A Multi-Scale Video Longformer Network for Action Recognition
Source abstract
Abstract
Action recognition has found extensive applications in fields such as video classification and security monitoring. However, existing action recognition methods, such as those based on 3D convolutional neural networks, often struggle to capture comprehensive global information. Meanwhile, transformer-based approaches face challenges associated with excessively high computational complexity. We introduce a Multi-Scale Video Longformer network (MSVL), built upon the 3D Longformer architecture featuring a “local attention + global features” attention mechanism, enabling us to reduce computational complexity while preserving global modeling capabilities. Specifically, MSVL gradually reduces the video feature resolution and increases the feature dimensions across four stages. In the lower layers of the network (stage 1, stage 2), we leverage local window attention to alleviate local redundancy and computational demands. Concurrently, global tokens are employed to retain global features. In the higher layers of the network (stage 3, stage 4), this local window attention evolves into a dense computation mechanism, enhancing overall performance. Finally, extensive experiments are conducted on UCF101 (97.6%), HMDB51 (72.9%), and the assembly action dataset (100.0%), demonstrating the effectiveness and efficiency of the MSVL.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Gait Recognition and Analysis · Hand Gesture Recognition Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Mechanical Engineering and Rail Transit, Changzhou University, Changzhou 213164, China
School of Innovation and Entrepreneurship, Changzhou University, Changzhou 213164, China
Funding
Jiangsu Carbon Peak Carbon Neutrality Science and Technology Innovation Project of China
References (44)
Show source-provided references
1. Yang, S., Zhao, Y., and Ma, Y. (2019, January 12–14). Analysis of the reasons and development of short video application-Taking Tik Tok as an example. Proceedings of the 2019 9th International Conference on Information and Social Science (ICISS 2019), Manila, Philippines.
2. Xiao, X., Xu, D., and Wan, W. (2016, January 11–12). Overview: Video recognition from handcrafted method to deep learning method. Proceedings of the 2016 International Conference on Audio, Language and Image Processing (ICALIP), Shanghai, China.10.1109/ICALIP.2016.7846652 ↗
3. Understanding video events: A survey of methods for automatic interpretation of semantic occurrences in video10.1109/TSMCC.2009.2023380 ↗
4. Dalal, N., Triggs, B., and Schmid, C. (2006, January 7–13). Human detection using oriented histograms of flow and appearance. Proceedings of the Computer Vision–ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria.10.1007/11744047_33 ↗
5. Wang, H., and Schmid, C. (2013, January 1–8). Action recognition with improved trajectories. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.441 ↗
6. O’Shea, K., and Nash, R. (2015). An introduction to convolutional neural networks. arXiv.
7. Attention is all you need
8. Spatio-temporal motion detection for intelligent surveillance applications10.1142/S0219876213500977 ↗
9. Two-stream convolutional networks for action recognition in videos
10. Huang, D.A., Ramanathan, V., Mahajan, D., Torresani, L., Paluri, M., Fei-Fei, L., and Niebles, J.C. (2018, January 18–23). What makes a video a video: Analyzing temporal information in video understanding models and datasets. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00769 ↗
11. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.510 ↗
12. Carreira, J., and Zisserman, A. (2017, January 21–26). Quo vadis, action recognition? A new model and the kinetics dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
13. Albawi, S., Mohammed, T.A., and Al-Zawi, S. (2017, January 21–23). Understanding of a convolutional neural network. Proceedings of the 2017 International Conference on Engineering and Technology (ICET), Antalya, Turkey.10.1109/ICEngTechnol.2017.8308186 ↗
14. A survey of convolutional neural networks: Analysis, applications, and prospects10.1109/TNNLS.2021.3084827 ↗
15. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
16. Bertasius, G., Wang, H., and Torresani, L. (2021, January 18–24). Is space-time attention all you need for video understanding?. Proceedings of the 2021 International Conference on Machine Learning, Online.
17. Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., and Feichtenhofer, C. (2021, January 11–17). Multiscale vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00675 ↗
18. Beltagy, I., Peters, M.E., and Cohan, A. (2020). Longformer: The long-document transformer. arXiv.
19. Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., and Van Gool, L. (2016, January 11–14). Temporal segment networks: Towards good practices for deep action recognition. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_2 ↗
20. Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., and Paluri, M. (2018, January 18–23). A closer look at spatiotemporal convolutions for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00675 ↗
21. Sun, C., Shrivastava, A., Vondrick, C., Murphy, K., Sukthankar, R., and Schmid, C. (2018, January 8–14). Actor-centric relation network. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01252-6_20 ↗
22. Wu, C.Y., Feichtenhofer, C., Fan, H., He, K., Krahenbuhl, P., and Girshick, R. (2019, January 15–20). Long-term feature banks for detailed video understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00037 ↗
23. Feichtenhofer, C., Fan, H., Malik, J., and He, K. (November, January 27). Slowfast networks for video recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
24. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–23). Non-local neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
25. Girdhar, R., Carreira, J., Doersch, C., and Zisserman, A. (2019, January 15–20). Video action transformer network. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00033 ↗
26. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C. (2021, January 11–17). Vivit: A video vision transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00676 ↗
27. Zhang, Y., Li, X., Liu, C., Shuai, B., Zhu, Y., Brattoli, B., Chen, H., Marsic, I., and Tighe, J. (2021, January 11–17). Vidtr: Video transformer without convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01332 ↗
28. End-to-end temporal action detection with transformer10.1109/TIP.2022.3195321 ↗
29. Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., and Hassner, T. (2022, January 23–27). An Efficient Spatio-Temporal Pyramid Transformer for Action Detection. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.
30. Wu, C.Y., Li, Y., Mangalam, K., Fan, H., Xiong, B., Malik, J., and Feichtenhofer, C. (2022, January 18–24). Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01322 ↗
31. Wei, C., Fan, H., Xie, S., Wu, C.Y., Yuille, A., and Feichtenhofer, C. (2022, January 18–24). Masked feature prediction for self-supervised visual pre-training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01426 ↗
32. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training
33. Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., and Hu, H. (2022, January 18–24). Video swin transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00320 ↗
34. Li, K., Wang, Y., Gao, P., Song, G., Liu, Y., Li, H., and Qiao, Y. (2022). Uniformer: Unified transformer for efficient spatiotemporal representation learning. arXiv.
35. Neimark, D., Bar, O., Zohar, M., and Asselmann, D. (2021, January 11–17). Video transformer network. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00355 ↗
36. Zhang, P., Dai, X., Yang, J., Xiao, B., Yuan, L., Zhang, L., and Gao, J. (2021, January 11–17). Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00299 ↗
37. Chu, X., Zhang, B., Tian, Z., Wei, X., and Xia, H. (2021). Do we really need explicit position encodings for vision transformers. arXiv.
38. Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv.
39. Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T. (2011, January 6–13). HMDB: A large video database for human motion recognition. Proceedings of the 2011 International Conference on Computer Vision, Washington, DC, USA.10.1109/ICCV.2011.6126543 ↗
40. Fan, H., Li, Y., Xiong, B., Lo, W.Y., and Feichtenhofer, C. (2024, January 23). PySlowFast. Available online: https://github.com/facebookresearch/slowfast.
41. Li, C., Yang, J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L., and Gao, J. (2021). Efficient self-supervised vision transformers for representation learning. arXiv.
42. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22–29). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.74 ↗
43. Contributors, M. (2024, January 23). OpenMMLab’s Next Generation Video Understanding Toolbox and Benchmark. Available online: https://github.com/open-mmlab/mmaction2.
44. Lin, J., Gan, C., and Han, S. (November, January 27). TSM: Temporal Shift Module for Efficient Video Understanding. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea.