Body-Pose-Guided Action Recognition with Convolutional Long Short-Term Memory (LSTM) in Aerial Videos
Source abstract
Abstract
The accurate detection and recognition of human actions play a pivotal role in aerial surveillance, enabling the identification of potential threats and suspicious behavior. Several approaches have been presented to address this problem, but the limitation still remains in devising an accurate and robust solution. To this end, this paper presents an effective action recognition framework for aerial surveillance, employing the YOLOv8-Pose keypoints extraction algorithm and a customized sequential ConvLSTM (Convolutional Long Short-Term Memory) model for classifying the action. We performed a detailed experimental evaluation and comparison on the publicly available Drone Action dataset. The evaluation and comparison of the proposed framework with several existing approaches on the publicly available Drone Action dataset demonstrate its effectiveness, achieving a very encouraging performance. The overall accuracy of the framework on three provided dataset splits is 74%, 80%, and 70%, with a mean accuracy of 74.67%. Indeed, the proposed system effectively captures the spatial and temporal dynamics of human actions, providing a robust solution for aerial action recognition.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Video Surveillance and Tracking Methods · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Mechatronics Engineering, National University of Sciences and Technology (NUST), Islamabad 44000, Pakistan
National Centre of Robotics and Automation, Islamabad 44000, Pakistan
Funding
Higher Education Commission of Pakistan and the National Centre of Robotics and Automation
References (35)
Show source-provided references
1. A secured distributed detection system based on IPFS and blockchain for industrial image and video data security10.1016/j.jpdc.2021.02.022 ↗
2. Towards the sustainable development of smart cities through mass video surveillance: A response to the COVID-19 pandemic10.1016/j.scs.2020.102582 ↗
3. Smart city as a smart service system: Human-computer interaction and smart city surveillance systems10.1016/j.chb.2021.106923 ↗
4. Human action recognition approaches with video datasets—A survey10.1016/j.knosys.2021.106995 ↗
5. Sultani, W., and Shah, M. (2021). Human Action Recognition in Drone Videos Using a Few Aerial Training Examples. arXiv, Available online: http://arxiv.org/abs/1910.10027.10.1016/j.cviu.2021.103186 ↗
6. Wang, X., Xian, R., Guan, T., de Melo, C.M., Nogar, S.M., Bera, A., and Manocha, D. (2023). AZTR: Aerial Video Action Recognition with Auto Zoom and Temporal Reasoning. arXiv, Available online: http://arxiv.org/abs/2303.01589.10.1109/ICRA48891.2023.10160564 ↗
7. Handcrafted localized phase features for human action recognition10.1016/j.imavis.2022.104465 ↗
8. El-Ghaish, H., Hussein, M., Shoukry, A., and Onai, R. (2018). Human Action Recognition Based on Integrating Body Pose, Part Shape, and Motion, IEEE Access.10.1109/ACCESS.2018.2868319 ↗
9. Human Action Recognition using 3D Convolutional Neural Networks with 3D Motion Cuboids in Surveillance Videos10.1016/j.procs.2018.07.059 ↗
10. 3DFCNN: Real-time action recognition using 3D deep neural networks with raw depth information10.1007/s11042-022-12091-z ↗
11. Real-time human action recognition using raw depth video-based recurrent neural networks10.1007/s11042-022-14075-5 ↗
12. Human action recognition using attention based LSTM network with dilated CNN features10.1016/j.future.2021.06.045 ↗
13. Two-stream transformer network for sensor-based human activity recognition10.1016/j.neucom.2022.09.099 ↗
14. Improved two-stream model for human action recognition10.1186/s13640-020-00501-x ↗
15. Graph Convolutional Neural Network for Human Action Recognition: A Comprehensive Survey10.1109/TAI.2021.3076974 ↗
16. A comparative review of graph convolutional networks for human skeleton-based action recognition10.1007/s10462-021-10107-y ↗
17. Yang, J., Dong, X., Liu, L., Zhang, C., Shen, J., and Yu, D. (2022, January 18–24). Recurring the Transformer for Video Action Recognition. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01367 ↗
18. Wang, X., Zhang, S., Qing, Z., Shao, Y., Zuo, Z., Gao, C., and Sang, N. (2021, January 10–17). OadTR: Online Action Detection with Transformers. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00747 ↗
19. Barekatain, M., Martí, M., Shih, H.-F., Murray, S., Nakayama, K., Matsuo, Y., and Prendinger, H. (2017, January 21–26). Okutama-Action: An Aerial View Video Dataset for Concurrent Human Action Detection. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.10.1109/CVPRW.2017.267 ↗
20. Liu, C., and Szirányi, T. (2021). Real-Time Human Detection and Gesture Recognition for On-Board UAV Rescue. Sensors, 21.10.3390/s21062180 ↗
21. Human activity recognition from UAV-captured video sequences10.1016/j.patcog.2019.107140 ↗
22. Perera, A.G., Law, Y.W., and Chahl, J. (2019). Drone-Action: An Outdoor Recorded Drone Video Dataset for Action Recognition. Drones, 3.10.3390/drones3040082 ↗
23. Cascading Pose Features with CNN-LSTM for Multiview Human Action Recognition10.3390/signals4010002 ↗
24. Yang, S.-H., Baek, D.-G., and Thapa, K. (2022). Semi-Supervised Adversarial Learning Using LSTM for Human Activity Recognition. Sensors, 22.10.3390/s22134755 ↗
25. Kumar, A., and Rawat, Y.S. (2022, January 18–24). End-to-End Semi-Supervised Learning for Video Action Detection. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01429 ↗
26. Human action recognition using two-stream attention based LSTM networks10.1016/j.asoc.2019.105820 ↗
27. Mathew, S., Subramanian, A., and Pooja, S. (2023). Human Activity Recognition Using Deep Learning Approaches: Single Frame CNN and Convolutional LSTM. arXiv.
28. Zhang, J., Bai, F., Zhao, J., and Song, Z. (2021, January 26–28). Multi-views Action Recognition on 3D ResNet-LSTM Framework. Proceedings of the 2021 IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE), Nanchang, China.10.1109/ICBAIE52039.2021.9389913 ↗
29. Reis, D., Kupec, J., Hong, J., and Daoudi, A. (2023). Real-Time Flying Object Detection with YOLOv8. arXiv, Available online: http://arxiv.org/abs/2305.09972.
30. Arif, S., Wang, J., Ul Hassan, T., and Fei, Z. (2019). 3D-CNN-Based Fused Feature Maps with LSTM Applied to Action Recognition. Future Internet, 11.10.3390/fi11020042 ↗
31. Mateus, B.C., Mendes, M., Farinha, J.T., and Cardoso, A.M. (2021). Anticipating Future Behavior of an Industrial Press Using LSTM Networks. Appl. Sci., 11.10.3390/app11136101 ↗
32. Khan, L., Amjad, A., Afaq, K.M., and Chang, H.-T. (2022). Deep Sentiment Analysis Using CNN-LSTM Architecture of English and Roman Urdu Text Shared in Social Media. Appl. Sci., 12.10.3390/app12052694 ↗
33. Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition10.1016/j.patcog.2021.108487 ↗
34. He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv, Available online: http://arxiv.org/abs/1512.03385.10.1109/CVPR.2016.90 ↗
35. Chen, S., Xu, K., Jiang, X., and Sun, T. (2022). Pyramid Spatial-Temporal Graph Transformer for Skeleton-Based Action Recognition. Appl. Sci., 12.10.3390/app12189229 ↗