Enhanced Spatial Stream of Two-Stream Network Using Optical Flow for Human Action Recognition
Source abstract
Abstract
Introduction: Convolutional neural networks (CNNs) have maintained their dominance in deep learning methods for human action recognition (HAR) and other computer vision tasks. However, the need for a large amount of training data always restricts the performance of CNNs. Method: This paper is inspired by the two-stream network, where a CNN is deployed to train the network by using the spatial and temporal aspects of an activity, thus exploiting the strengths of both networks to achieve better accuracy. Contributions: Our contribution is twofold: first, we deploy an enhanced spatial stream, and it is demonstrated that models pre-trained on a larger dataset, when used in the spatial stream, yield good performance instead of training the entire model from scratch. Second, a dataset augmentation technique is presented to minimize overfitting of CNNs, where we increase the dataset size by performing various transformations on the images such as rotation and flipping, etc. Results: UCF101 is a standard benchmark dataset for action videos, and our architecture has been trained and validated on it. Compared with the other two-stream networks, our results outperformed them in terms of accuracy.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Anomaly Detection Techniques and Applications · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer and Software Engineering, College of Electrical and Mechanical Engineering, National University of Sciences and Technology, Islamabad 46000, Pakistan
School of Computer Science, University of Lincoln, Lincoln LN6 7TS, UK
Department of Computer Engineering, College of Computer and Information Sciences, King Saud University, P.O. Box 51178, Riyadh 11543, Saudi Arabia
Funding
King Saud University
References (33)
Show source-provided references
1. Human Activity Analysis: A Review10.1145/1922649.1922653 ↗
2. Understanding Video Events: A Survey of Methods for Automatic Interpretation of Semantic Occurrences in Video10.1109/TSMCC.2009.2023380 ↗
3. IoTAuth: IoT Sensor Data Analytics for User Authentication Using Discriminative Feature Analysis10.1109/ACCESS.2022.3178635 ↗
4. HMR-vid: A comparative analytical survey on human motion recognition in video data10.1007/s11042-020-09485-2 ↗
5. Qiu, Z., Yao, T., and Mei, T. (2017, January 22–29). Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.590 ↗
6. 3D Convolutional neural networks for human action recognition10.1109/TPAMI.2012.59 ↗
7. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–22). Non-local Neural Networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
8. Zhao, Y., Xiong, Y., and Lin, D. (2018, January 3–8). Trajectory convolution for action recognition. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
9. A survey of video datasets for human action and activity recognition10.1016/j.cviu.2013.01.013 ↗
10. EDCAR: A knowledge representation framework to enhance automatic video surveillance10.1016/j.eswa.2019.04.031 ↗
11. High-Level Video Event Modeling, Recognition, and Reasoning via Petri Net10.1109/ACCESS.2019.2936493 ↗
12. Action Recognition from Arbitrary Views Using Transferable Dictionary Learning10.1109/TIP.2018.2836323 ↗
13. Simonyan, K., and Zisserman, A. (2014, January 8–13). Two-Stream Convolutional Networks for Action Recognition in Videos. Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
14. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014, January 23–28). Large-Scale Video Classification with Convolutional Neural Networks. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.223 ↗
15. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 11–18). Learning spatiotemporal features with 3D convolutional networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.510 ↗
16. Taylor, G.W., Fergus, R., LeCun, Y., and Bregler, C. (2010). Computer Vision–ECCV 2010, Proceedings of the 11th European Conference on Computer Vision, Heraklion, Crete, Greece, 5–11 September 2010, Springer.
17. Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection10.1016/j.cirp.2016.04.072 ↗
18. Long-Term Recurrent Convolutional Networks for Visual Recognition and Description10.1109/TPAMI.2016.2599174 ↗
19. Li, H., Chen, J., Hu, R., Yu, M., Chen, H., and Xu, Z. (2019). MultiMedia Modeling, Proceedings of the 25th International Conference, MMM 2019, Thessaloniki, Greece, 8–11 January 2019, Springer.
20. Ng, J., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., and Toderici, G. (2015, January 7–12). Beyond short snippets: Deep networks for video classification. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
21. Wang, H., and Schmid, C. (2013, January 1–8). Action recognition with improved trajectories. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.441 ↗
22. Feichtenhofer, C., Pinz, A., and Wildes, R.P. (2016, January 5–10). Spatiotemporal residual networks for video action recognition. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.10.1109/CVPR.2017.787 ↗
23. Feichtenhofer, C., Pinz, A., and Zisserman, A. (2016, January 27–30). Convolutional Two-Stream Network Fusion for Video Action Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.213 ↗
24. Going deeper with two-stream ConvNets for action recognition in video surveillance10.1016/j.patrec.2017.08.015 ↗
25. Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild. arXiv.
26. Schuldt, C., Laptev, I., and Caputo, B. (2004, January 26). Recognizing human actions: A local SVM approach. Proceedings of the 17th International Conference on Pattern Recognition, Cambridge, UK.10.1109/ICPR.2004.1334462 ↗
27. Shi, L., Zhang, Y., Cheng, J., and Lu, H. (2019, January 16–20). Two-stream adaptive graph convolutional networks for skeleton-based action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01230 ↗
28. MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object Segmentation10.1109/TIP.2020.3013162 ↗
29. Rethinking motion representation: Residual frames with 3D convnets10.1109/TIP.2021.3124156 ↗
30. ImageNet Large Scale Visual Recognition Challenge10.1007/s11263-015-0816-y ↗
31. A Survey on Transfer Learning10.1109/TKDE.2009.191 ↗
32. ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
33. Visualizing and understanding graph convolutional network10.1007/s11042-020-09885-4 ↗