Action Recognition and Detection Based on Deep Learning: A Comprehensive Summary
Source abstract
Abstract
Action recognition and detection is an important research topic in computer vision, which can be divided into action recognition and action detection. At present, the distinction between action recognition and action detection is not clear, and the relevant reviews are not comprehensive. Thus, this paper summarized the action recognition and detection methods and datasets based on deep learning to accurately present the research status in this field. Firstly, according to the way that temporal and spatial features are extracted from the model, the commonly used models of action recognition are divided into the two stream models, the temporal models, the spatiotemporal models and the transformer models according to the architecture. And this paper briefly analyzes the characteristics of the four models and introduces the accuracy of various algorithms in common data sets. Then, from the perspective of tasks to be completed, action detection is further divided into temporal action detection and spatiotemporal action detection, and commonly used datasets are introduced. From the perspectives of the two-stage method and one-stage method, various algorithms of temporal action detection are reviewed, and the various algorithms of spatiotemporal action detection are summarized in detail. Finally, the relationship between different parts of action recognition and detection is discussed, the difficulties faced by the current research are summarized in detail, and future development was prospected.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (100+)
Show source-provided references
1. Sika deer behavior recognition based on machine vision10.32604/cmc.2022.027457 ↗
2. S. Zhang, “Research on human action detection and recognition in videos,” Ph.D. dissertation, Huazhong University of Science and Technology, China, 2019.
3. A critical review of action recognition benchmarks
4. Review of video action recognition
5. Review of video action recognition method based on the depth of learning
6. Survey on human action recognition based on deep learning
7. Y. Zhu, X. Li, C. Liu, M. Zolfaghari and M. Li, “A comprehensive study of deep video action recognition,” arXiv preprint arXiv:2012.06567, 2020.
8. Human action recognition from various data modalities: A review
9. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn and N. Houlsby, “An image is worth 16 × 16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
10. A survey of advances in vision-based vehicle re-identification10.1016/j.cviu.2019.03.001 ↗
11. Learning spatiotemporal features with 3D convolutional networks
12. Long-term recurrent convolutional networks for visual recognition and description
13. Action recognition with improved trajectories
14. Two-stream convolutional networks for action recognition in videos
15. Learning spatio-temporal representation with pseudo-3d residual networks
16. VideoLSTM convolves, attends and flows for action recognition10.1016/j.cviu.2017.10.011 ↗
17. A. Diba, M. Fayyaz, V. Sharma, A. H. Karami and R. Yousefzadeh, “Temporal 3D convents: New architecture and transfer learning for video classification,” arXiv preprint arXiv:1711.08200, 2017.
18. Convolutional two-stream network fusion for video action recognition
19. Hidden two-stream convolutional networks for action recognition
20. Spatiotemporal residual networks for video action recognition
21. Temporal segment networks: Towards good practices for deep action recognition
22. Quo vadis, action recognition? A new model and the kinetics dataset
23. Learning spatio-temporal representation with local and global diffusion
24. Hallucinating IDT descriptors and I3D optical flow features for action recognition with CNNs
25. Late temporal modeling in 3D CNN architectures with BERT for action recognition
26. P. Koniusz, L. Wang and K. Sun, “High-order tensor pooling with attention for action recognition,” arXiv preprint arXiv:2110.05216, 2021.
27. Recognizing human actions: A local SVM approach
28. UCF101: A dataset of 101 human actions classes from videos in the wild
29. HMDB: A large video database for human motion recognition
30. A novel mid-level distinctive feature learning for action recognition via diffusion map10.1016/j.neucom.2016.08.057 ↗
31. Actions as space-time shapes
32. Action recognition new framework with robust 3D-TCCHOGAC and 3D-HOOFGAC10.1007/s11042-016-3279-4 ↗
33. Actions in context
34. Y. Wang, V. Tran and M. Hoai, “Evolution-preserving dense trajectory descriptors,” arXiv preprint arXiv:1702.04037, 2017.
35. Modeling temporal structure of decomposable motion segments for activity classification
36. VLAD3: Encoding dynamics of deep features for action recognition
37. Multiview transformers for video recognition
38. W. Kay, J. Carreira, K. Simonyan, B. Zhang and A. Zisserman, “The kinetics human action video dataset,” arXiv preprint arXiv:1705.06950, 2017.
39. The “Something Something” video database for learning and evaluating visual common sense
40. UniFormer: Unified transformer for efficient spatial-temporal representation learning
41. J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier and A. Zisserman, “A short note about Kinetics-600,” arXiv preprint arXiv:1808.01340, 2018.
42. Merlot reserve: Neural script knowledge through vision and language and sound
43. Z. Tong, Y. Song, J. Wang and L. Wang, “Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training,” arXiv preprint arXiv:2203.12602, 2022.
44. J. Carreira, E. Noland, C. Hillier and A. Zisserman, “A short note on the kinetics-700 human action dataset,” arXiv preprint arXiv:1907.06987, 2019.
45. Multiview transformers for video recognition
46. M. Cheng, K. Cai and M. Li, “RWF-2000: An open large scale video database for violence detection,” arXiv preprint arXiv:1911.05913, 2019.
47. Violence behavior recognition of two-cascade temporal shift module with attention mechanism10.1117/1.JEI.30.4.043009 ↗
48. The recognition of human movement using temporal templates
49. Effective 3D action recognition using EigenJoints10.1016/j.jvcir.2013.03.001 ↗
50. An efficient dense and scale-invariant spatio-temporal interest point detector
51. Dense trajectories and motion boundary descriptors for action recognition10.1007/s11263-012-0594-8 ↗
52. ImageNet classification with deep convolutional neural networks
53. Spatiotemporal residual networks for video action recognition
54. Deep residual learning for image recognition
55. Beyond short snippets: Deep networks for video classification
56. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
57. Slowfast networks for video recognition
58. Gate-shift networks for video action recognition
59. Convolutional LSTM network: A machine learning approach for precipitation nowcasting
60. I3D-LSTM: A new model for human action recognition
61. 3D convolutional neural networks for human action recognition10.1109/TPAMI.2012.59 ↗
62. Densely connected convolutional networks
63. Going deeper with convolutions
64. TSM: Temporal shift module for efficient video understanding
65. Temporal interlacing network
66. RubiksNet: Learnable 3D-shift for efficient video action recognition
67. TEA: Temporal excitation and aggregation for action recognition
68. VIVIT: A video vision transformer
69. Attention-based LSTM network for action recognition in sports
70. Keeping your eye on the ball: Trajectory attention in video transformers
71. Direcformer: A directed attention in transformer approach to robust action recognition
72. Multiview transformers for video recognition
73. Self-supervised video transformer
74. Recurring the transformer for video action recognition
75. Learning to localize actions from moments
76. Crucianu · MEXaction2: Action Detection and Localization Dataset · 2015
77. Dynamic temporal pyramid network: A closer look at multi-scale modeling for activity detection
78. FineAction: A fine-grained video dataset for temporal action localization10.1109/TIP.2022.3217368 ↗
79. Unsupervised deep metric learning via orthogonality based probabilistic loss
80. Two-stream collaborative learning with spatial-temporal attention for video classification
81. AVA: A video dataset of spatio-temporally localized atomic visual actions
82. Multisports: A multi-person video dataset of spatio-temporally localized sports actions
83. R-C3D: Region convolutional 3d network for temporal activity detection
84. G-TAD: Sub-graph localization for temporal action detection
85. Rethinking the faster R-CNN architecture for temporal action localization
86. 3C-NET: Category count and center loss for weakly-supervised action localization
87. BSN: Boundary sensitive network for temporal action proposal generation
88. Graph convolutional networks for temporal action localization
89. Temporal action detection with structured segment networks
90. BMN: Boundary-matching network for temporal action proposal generation
91. Actionformer: Localizing moments of actions with transformers
92. X. Wang, Z. Qing, Z. Huang, Y. Feng and N. Sang, “Proposal relation network for temporal action detection,” arXiv preprint arXiv:2106.11812, 2021.
93. Temporal action localization in untrimmed videos via multi-stage CNNs
94. CDC: Convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos
95. Turn tap: Temporal unit regression network for temporal action proposals
96. Stacked LSTM network for human activity recognition using smartphone data
97. R-C3D: Region convolutional 3D network for temporal activity detection
98. J. Gao, Z. Yang and R. Nevatia, “Cascaded boundary regression for temporal action detection,” arXiv preprint arXiv:1705.01180, 2017.10.5244/C.31.52 ↗
99. Multi-granularity generator for temporal action proposal
100. React: Temporal action detection with relational queries