Human Action Recognition Based on Improved Two-Stream Convolution Network
Source abstract
Abstract
Two-stream convolution network (2SCN) is a classical method of action recognition. It is capable of extracting action information from two dimensions: spatial and temporal streams. However, the method of extracting motion features from a spatial stream is single-frame recognition, and there is still room for improvement in the perception ability of appearance coherence features. The classical two-stream convolution network structure is modified in this paper by utilizing the strong mining capabilities of the bidirectional gated recurrent unit (BiGRU) to allow the neural network to extract the appearance coherence features of actions. In addition, this paper introduces an attention mechanism (SimAM) based on neuroscience theory, which improves the accuracy and stability of neural networks. Experiments show that the method proposed in this paper (BS-2SCN, BiGRU-SimAM Two-stream convolution network) has high accuracy. The accuracy is improved by 2.6% on the UCF101 data set and 11.7% on the HMDB51 data set.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Anomaly Detection Techniques and Applications · Gait Recognition and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Automation, Nanjing University of Information Science & Technology, Nanjing 210044, China
Department of Electrical Engineering & Electronics, University of Liverpool, Liverpool L69 7ZX, UK
Jiangsu Provincial Collaborative Innovation Center for Atmospheric Environment and Equipment Technology, Nanjing University of Information Science & Technology, Nanjing 210044, China
Funding
NUIST Students’ Platform for Innovation and Entrepreneurship Training Program
References (74)
Show source-provided references
1. Human-Exploratory-Procedure-Based Hybrid Measurement Fusion for Material Recognition10.1109/TMECH.2021.3080378 ↗
2. A Target Grabbing Strategy for Telerobot Based on Improved Stiffness Display Device10.1109/JAS.2016.7510256 ↗
3. An Appearance-Based Representation of Action10.1109/ICPR.1996.546039 ↗
4. Free Viewpoint Action Recognition Using Motion History Volumes10.1016/j.cviu.2006.07.013 ↗
5. Laptev, I., Marszalek, M., Schmid, C., and Rozenfeld, B. (2008, January 23–28). Learning Realistic Human Actions from Movies. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.10.1109/CVPR.2008.4587756 ↗
6. Dense Trajectories and Motion Boundary Descriptors for Action Recognition10.1007/s11263-012-0594-8 ↗
7. Wang, H., and Schmid, C. (2013, January 1–8). Action Recognition with Improved Trajectories. Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.441 ↗
8. Hu, K., Ding, Y., Jin, J., Weng, L., and Xia, M. (2022). Skeleton Motion Recognition Based on Multi-Scale Deep Spatio-Temporal Features. Appl. Sci., 12.10.3390/app12031028 ↗
9. Effective 3D Action Recognition Using EigenJoints10.1016/j.jvcir.2013.03.001 ↗
10. Liu, X., Chen, H.-X., and Liu, B.-Y. (2022). Dynamic Anchor: A Feature-Guided Anchor Strategy for Object Detection. Appl. Sci., 18.10.3390/app12104897 ↗
11. Hu, K., Tian, L., Weng, C., Weng, L., Zang, Q., Xia, M., and Qin, G. (2021). Data-Driven Control Algorithm for Snake Manipulator. Appl. Sci., 11.10.3390/app11178146 ↗
12. Multi-scale strip pooling feature aggregation network for cloud and cloud shadow segmentation10.1007/s00521-021-06802-0 ↗
13. MFAGCN: A New Framework for Identifying Power Grid Branch Parameters10.1016/j.epsr.2022.107855 ↗
14. MLNet: Multichannel feature fusion lozenge network for land segmentation10.1117/1.JRS.16.016513 ↗
15. ImageNet Classification with Deep Convolutional Neural Networks10.1145/3065386 ↗
16. Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
17. Szegedy, C., Wei, L., Yangqing, J., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going Deeper with Convolutions. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
18. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
19. 3D Convolutional Neural Networks for Human Action Recognition10.1109/TPAMI.2012.59 ↗
20. Simonyan, K., and Zisserman, A. (2014). Two-Stream Convolutional Networks for Action Recognition in Videos. arXiv.
21. Bahdanau, D., Cho, K., and Bengio, Y. (2016). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv.
22. Luong, M.-T., Pham, H., and Manning, C.D. (2015). Effective Approaches to Attention-Based Neural Machine Translation. arXiv.10.18653/v1/D15-1166 ↗
23. Hu, J., Shen, L., and Sun, G. (2015). Squeeze-and-Excitation Networks. 10. arXiv.
24. CBAM: Convolutional Block Attention Module10.1007/978-3-030-01234-2_1 ↗
25. PADANet: Parallel asymmetric double attention network for clouds and its shadow detection10.1117/1.JRS.15.046512 ↗
26. SUACDNet: Attentional change detection network based on siamese U-shaped structure
27. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
28. Distributed Representations, Simple Recurrent Networks, and Grammatical Structure10.1007/BF00114844 ↗
29. (1997). Hochreiter S; Schmidhuber J Long Short-Term Memory. Neural Comput., 9, 1735–1780.10.1162/neco.1997.9.8.1735 ↗
30. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. arXiv.10.3115/v1/D14-1179 ↗
31. Chakrabarty, A., Pandit, O.A., and Garain, U. (2017). Context Sensitive Lemmatization Using Two Successive Bidirectional Gated Recurrent Networks. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics.10.18653/v1/P17-1136 ↗
32. Yang, L., Zhang, R.-Y., Li, L., and Xie, X. (2021, January 18–24). SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks. Proceedings of the 38th International Conference on Machine Learning, Online.
33. Allport, A. (1989). Visual Attention. Foundations of Cognitive Science, The MIT Press.10.7551/mitpress/3072.003.0018 ↗
34. Cheng, X., Li, X., Yang, J., and Tai, Y. (2018, January 20–24). SESR: Single Image Super Resolution with Recursive Squeeze and Excitation Networks. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China.10.1109/ICPR.2018.8546130 ↗
35. Delving Deep into Spatial Pooling for Squeeze-and-Excitation Networks10.1016/j.patcog.2021.108159 ↗
36. ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification
37. Improving Transfer Learning and Squeeze- and-Excitation Networks for Small-Scale Fine-Grained Fish Image Classification10.1109/ACCESS.2018.2885055 ↗
38. Automated Pulmonary Nodule Detection in CT Images Using 3D Deep Squeeze-and-Excitation Networks10.1007/s11548-019-01979-1 ↗
39. Combining 3D-CNN and Squeeze-and-Excitation Networks for Remote Sensing Sea Ice Image Classification
40. PRI Modulation Recognition Based on Squeeze-and-Excitation Networks10.1109/LCOMM.2020.2970397 ↗
41. Strip pooling channel spatial attention network for the segmentation of cloud and cloud shadow10.1016/j.cageo.2021.104940 ↗
42. Hou, Q., Zhou, D., and Feng, J. (2021, January 20–25). Coordinate Attention for Efficient Mobile Network Design. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.01350 ↗
43. A Convolutional Attention Residual Network for Stereo Matching10.1109/ACCESS.2020.2980243 ↗
44. CBAM-GAN: Generative Adversarial Networks Based on Convolutional Block Attention Module10.1007/978-3-030-24274-9_20 ↗
45. Wang, S.-H., Fernandes, S., Zhu, Z., and Zhang, Y.-D. (2021). AVNC: Attention-Based VGG-Style Network for COVID-19 Diagnosis by CBAM. IEEE Sens. J.10.1109/JSEN.2021.3062442 ↗
46. Cropping and Attention Based Approach for Masked Face Recognition10.1007/s10489-020-02100-9 ↗
47. Facial Expression Recognition via a CBAM Embedded Network10.1016/j.procs.2020.06.115 ↗
48. Fu, H., Song, G., and Wang, Y. (2021). Improved YOLOv4 Marine Target Detection Combined with CBAM. Symmetry, 13.10.3390/sym13040623 ↗
49. A New Attention-Based CNN Approach for Crop Mapping Using Time Series Sentinel-2 Images10.1016/j.compag.2021.106090 ↗
50. Attention Receptive Pyramid Network for Ship Detection in SAR Images10.1109/JSTARS.2020.2997081 ↗
51. Visual Attention: The Past 25 Years10.1016/j.visres.2011.04.012 ↗
52. (2022). IL-MCAM: An Interactive Learning and Multi-Channel Attention Mechanism-Based Weakly Supervised Colorectal Histopathology Image Classification Approach. Comput. Biol. Med., 143, 105265.10.1016/j.compbiomed.2022.105265 ↗
53. Xie, J., Wu, Z., Zhu, R., and Zhu, H. (2021, January 15). Melanoma Detection Based on Swin Transformer and SimAM. Proceedings of the 2021 IEEE 5th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Xi’an, China.10.1109/ITNEC52019.2021.9587071 ↗
54. An Efficient Extreme-Exposure Image Fusion Method10.1088/1742-6596/2137/1/012061 ↗
55. Hu, K., Zheng, F., Weng, L., Ding, Y., and Jin, J. (2021). Action Recognition Algorithm of Spatio–Temporal Differential LSTM Based on Feature Enhancement. Appl. Sci., 11.10.3390/app11177876 ↗
56. Miao, S., Xia, M., Qian, M., Zhang, Y., Liu, J., and Lin, H. (2022). Cloud/shadow segmentation based on multi-level feature enhanced network for remote sensing imagery. Int. J. Remote Sens., 1–21.10.1080/01431161.2021.2014077 ↗
57. Early and Late Mechanisms of Surround Suppression in Striate Cortex of Macaque10.1523/JNEUROSCI.3414-05.2005 ↗
58. Actions as Space-Time Shapes10.1109/TPAMI.2007.70711 ↗
59. Rodriguez, M.D., Ahmed, J., and Shah, M. (2008, January 23–28). Action MACH a Spatio-Temporal Maximum Average Correlation Height Filter for Action Recognition. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.10.1109/CVPR.2008.4587727 ↗
60. Liu, J., Luo, J., and Shah, M. (2009, January 22–24). Recognizing Realistic Actions from Videos “in the Wild”. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206744 ↗
61. Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T. (2011, January 6–13). HMDB: A Large Video Database for Human Motion Recognition. Proceedings of the 2011 International Conference on Computer Vision, Washington, DC, USA.10.1109/ICCV.2011.6126543 ↗
62. Recognizing 50 Human Action Categories of Web Videos10.1007/s00138-012-0450-4 ↗
63. Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv.
64. Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., and Natsev, P. (2017). The Kinetics Human Action Video Dataset. arXiv.
65. Multi-stage feature constraints learning for age estimation10.1109/TIFS.2020.2969552 ↗
66. Diba, A., Sharma, V., and Van Gool, L. (2017, January 21–26). Deep Temporal Linear Encoding Networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.168 ↗
67. Zhou, Y., Sun, X., Zha, Z.-J., and Zeng, W. (2018, January 18–23). MiCT: Mixed 3D/2D Convolutional Tube for Human Action Recognition. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00054 ↗
68. Wang, X., Farhadi, A., and Gupta, A. (2016, January 27–30). Actions Transformations. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.291 ↗
69. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014, January 23–28). Large-Scale Video Classification with Convolutional Neural Networks. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Washington, DC, USA.10.1109/CVPR.2014.223 ↗
70. Marszalek, M., Laptev, I., and Schmid, C. (2009, January 20–25). Actions in Context. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPRW.2009.5206557 ↗
71. Qiu, Z., Yao, T., and Mei, T. (2017, January 22–29). Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.590 ↗
72. Ng, J.Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., and Toderici, G. (2015, January 7–12). Beyond Short Snippets: Deep Networks for Video Classification. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
73. Feichtenhofer, C., Pinz, A., and Zisserman, A. (2016, January 27–30). Convolutional Two-Stream Network Fusion for Video Action Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.213 ↗
74. Zhu, Y., Lan, Z., Newsam, S., and Hauptmann, A.G. (2018). Hidden Two-Stream Convolutional Networks for Action Recognition. arXiv.