Learning spatial–temporal features via a pose-flow relational model for action recognition
Source abstract
Abstract
Pose-based action recognition has always been an important research field in computer vision. However, most existing pose-based methods are built upon human skeleton data, which cannot be used to exploit the feature of the motion-related object, i.e., a crucial clue of discriminating human actions. To address this issue, we propose a novel pose-flow relational model, which can benefit from both pose dynamics and optical flow. First, we introduce a pose estimation module to extract the skeleton data of the key person from the raw video. Second, a hierarchical pose-based network is proposed to effectively explore the rich spatial–temporal features of human skeleton positions. Third, we embed an inflated 3D network to capture the subtle cues of the motion-related object from optical flow. Additionally, we evaluate our model on four popular action recognition benchmarks (HMDB-51, JHMDB, sub-JHMDB, and SYSU 3D). Experimental results demonstrate that the proposed model outperforms the existing pose-based methods in human action recognition.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Gait Recognition and Analysis · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Nanjing Tech University 1 , Nanjing 210000, China
School of Information and Control Engineering, China University of Mining and Technology 2 , Xuzhou 221000, China
Funding
Natural Science Foundation of Jiangsu Higher Education Institutions of China · China Postdoctoral Science Foundation · National Natural Science Foundation of China
References (50)
Show source-provided references
1. Abnormal event detection via covariance matrix for optical flow based feature10.1007/s11042-017-5309-2 ↗
2. Hierarchical graphical-based human pose estimation via local multi-resolution convolutional neural network10.1063/1.5024463 ↗
3. Human pose estimation with multiple mixture parts model based on upper body categories10.1117/1.jei.24.4.043021 ↗
4. Recognizing facial expressions of occluded faces using convolutional neural networks
5. Combining local appearance and holistic view: Dual-source deep neural networks for human pose estimation
6. Deepercut: A deeper, stronger, and faster multi-person pose estimation model
7. RMPE: Regional multi-person pose estimation
8. Realtime multi-person 2D pose estimation using part affinity fields
9. Moving-object tracking algorithm based on PCA-SIFT and optimization for underground coal mines10.1109/access.2019.2899362 ↗
10. Human action recognition: Pose-based attention draws focus to hands
11. 3D convolutional neural networks for human action recognition10.1109/tpami.2012.59 ↗
12. Large-scale video classification with convolutional neural networks
13. Learning spatiotemporal features with 3D convolutional networks
14. Quo vadis, action recognition? A new model and the kinetics dataset
15. HMDB: A large video database for human motion recognition
16. Towards understanding action recognition
17. Jointly learning heterogeneous features for RGB-D activity recognition
18. Beyond physical connections: Tree models in human pose estimation
19. Deeppose: Human pose estimation via deep neural networks
20. P-CNN: Pose-based CNN features for action recognition
21. A hierarchical pose-based approach to complex action understanding using dictionaries of actionlets and motion poselets
22. An end-to-end spatio-temporal attention model for human action recognition from skeleton data
23. Pose-based multisource networks using convolutional neural network and long short-term memory for action recognition10.1117/1.jei.28.4.043018 ↗
24. 2D/3D pose estimation and action recognition using multitask deep learning
25. Human action recognition: Pose-based attention draws focus to hands
26. Multi-modal three-stream network for action recognition
27. TensorFlow: A system for large-scale machine learning
28. A duality based approach for realtime TV-L1 optical flow
29. Two-stream convolutional networks for action recognition in videos
30. Action recognition with trajectory-pooled deep-convolutional descriptors
31. Convolutional two-stream network fusion for video action recognition
32. UntrimmedNets for weakly supervised action recognition and detection
33. Chained multi-stream networks exploiting pose, motion, and appearance for action classification and detection
34. Spatiotemporal residual networks for video action recognition
35. Video representation learning using discriminative pooling
36. Optical flow guided feature: A fast and robust motion representation for video action recognition
37. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification
38. A closer look at spatiotemporal convolutions for action recognition
39. PoTion: Pose motion representation for action recognition
40. PA3D: Pose-action 3D machine for video recognition
41. Spatio-temporal LSTM with trust gates for 3D human action recognition
42. View adaptive recurrent neural networks for high performance human action recognition from skeleton data
43. Global context-aware attention lstm network for 3D action recognition
44. Memory attention networks for skeleton-based action recognition
45. Finding action tubes
46. Cross-view action modeling, learning and recognition
47. Joint action recognition and pose estimation from video
48. RPAN: An end-to-end recurrent pose-attention network for action recognition in videos
49. Action recognition with joints-pooled 3D deep convolutional descriptors
50. Part-aligned pose-guided recurrent network for action recognition10.1016/j.patcog.2019.03.010 ↗