Video Description Model Based on Temporal-Spatial and Channel Multi-Attention Mechanisms
Source abstract
Abstract
Video description plays an important role in the field of intelligent imaging technology. Attention perception mechanisms are extensively applied in video description models based on deep learning. Most existing models use a temporal-spatial attention mechanism to enhance the accuracy of models. Temporal attention mechanisms can obtain the global features of a video, whereas spatial attention mechanisms obtain local features. Nevertheless, because each channel of the convolutional neural network (CNN) feature maps has certain spatial semantic information, it is insufficient to merely divide the CNN features into regions and then apply a spatial attention mechanism. In this paper, we propose a temporal-spatial and channel attention mechanism that enables the model to take advantage of various video features and ensures the consistency of visual features between sentence descriptions to enhance the effect of the model. Meanwhile, in order to prove the effectiveness of the attention mechanism, this paper proposes a video visualization model based on the video description. Experimental results show that, our model has achieved good performance on the Microsoft Video Description (MSVD) dataset and a certain improvement on the Microsoft Research-Video to Text (MSR-VTT) dataset.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Human Pose and Action Recognition · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Funding
National Key Research and Development Program · Sichuan Province Scientific and Technological Support Project
References (38)
Show source-provided references
1. Two-stream multi-rate recurrent neural network for video-based pedestrian re-identification10.1109/TII.2017.2767557 ↗
2. Park, J.S., Rohrbach, M., Darrell, T., and Rohrbach, A. (2018). Adversarial Inference for Multi-Sentence Video Description. arXiv.10.1109/CVPR.2019.00676 ↗
3. Kim, B., Shin, S., and Jung, H. (2019). Variational Autoencoder-Based Multiple Image Captioning Using a Caption Attention Map. Appl. Sci., 9.10.3390/app9132699 ↗
4. Zhang, J., and Peng, Y. (2019). Object-aware Aggregation with Bidirectional Temporal Graph for Video Captioning. arXiv.10.1109/CVPR.2019.00852 ↗
5. Arachchi, S.P.K., Shih, T.K., and Hakim, N.L. (2020). Modelling a Spatial-Motion Deep Learning Framework to Classify Dynamic Patterns of Videos. Appl. Sci., 10.10.3390/app10041479 ↗
6. Recurrent convolutional video captioning with global and local attention10.1016/j.neucom.2019.08.042 ↗
7. Video captioning by adversarial LSTM10.1109/TIP.2018.2855422 ↗
8. CAM-RNN: Co-Attention Model based RNN for Video Captioning10.1109/TIP.2019.2916757 ↗
9. Video captioning using boosted and parallel Long Short-Term Memory networks10.1016/j.cviu.2019.102840 ↗
10. Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., and Saenko, K. (2014). Translating videos to natural language using deep recurrent neural networks. arXiv.10.3115/v1/N15-1173 ↗
11. Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., and Saenko, K. (2015, January 7–13). Sequence to sequence-video to text. Proceedings of the IEEE Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.515 ↗
12. Xu, K., Ba, J., Kiros, R., Courville, A., Salakhtdinov, R., Zemel, R., and Bengio, Y. (2015, January 6–11). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the International Conference on Machine Learning, Lille, France.
13. Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., and Courville, A. (2015, January 7–13). Describing videos by exploiting temporal structure. Proceedings of the IEEE Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.512 ↗
14. Automatic fruits classification using deep learning for industrial applications10.1109/TII.2018.2875149 ↗
15. Xiao, T., Xu, Y., Yang, K., Zhang, J., Peng, Y., and Zhang, Z. (2015, January 7–12). The application of two-level attention models in deep convolutional neural network for fine-grained image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
16. Papineni, K. (2002, January 6–13). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, In Proceedings of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
17. Vedantam, R., Lawrence, C., and Parikh, D. (2015, January 7–12). Cider: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
18. Denkowski, M., and Lavie, A. (2014, January 26–27). Meteor universal: Language specific translation evaluation for any target language. Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, MD, USA.10.3115/v1/W14-3348 ↗
19. Lin, C.Y. (2004, January 25–26). Rouge: A package for automatic evaluation of summaries. Proceedings of the Text Summarization Branches Out: ACL-04 Workshop, Barcelona, Spain.
20. Zhou, L., Kalantidis, Y., Chen, X., Corso, J.J., and Rohrbach, M. (2018). Grounded Video Description. arXiv.10.1109/CVPR.2019.00674 ↗
21. Pei, W., Zhang, J., Wang, X., Ke, L., Shen, X., and Tai, Y.W. (2019, January 16–20). Memory-Attended Recurrent Network for Video Captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00854 ↗
22. Wu, A., Han, Y., Yang, Y., Hu, Q., and Wu, F. (2019). Convolutional Reconstruction-to-Sequence for Video Captioning. IEEE Trans. Circuits Syst. Video Technol.10.1109/TCSVT.2019.2956593 ↗
23. Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., and Courville, A. (2015). Video description generation incorporating spatio-temporal features and a soft-attention mechanism. arXiv.
24. Zeiler, M.D., and Fergus, R. (2014). Visualizing and understanding convolutional networks. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014, Springer.10.1007/978-3-319-10590-1_53 ↗
25. Chen, D.L., and Dolan, W.B. (2011, January 19–24). Collecting highly parallel data for paraphrase evaluation. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, Portland, OR, USA.
26. Xu, J., Mei, T., Yao, T., and Rui, Y. (2016, January 27–30). MSR-VTT: A Large Video Description Dataset for Bridging Video and Language. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.571 ↗
27. Bird, S., and Loper, E. (2004, January 21–26). NLTK: The Natural Language Toolkit. Proceedings of the ACL Interactive Poster and Demonstration Sessions, Barcelona, Spain.10.3115/1219044.1219075 ↗
28. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
29. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (July, January 26). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
30. Kingma, D., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
31. Pan, Y., Mei, T., Yao, T., Li, H., and Rui, Y. (2016, January 27–30). Jointly modeling embedding and translation to bridge video and language. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.497 ↗
32. Pan, P., Xu, Z., Yang, Y., Wu, F., and Zhuang, Y. (2016, January 27–30). Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.117 ↗
33. Yu, H., Wang, J., Huang, Z., Yang, Y., and Xu, W. (2016, January 27–30). Video paragraph caption using hierarchical recurrent neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.496 ↗
34. Zhang, X., Gao, K., Zhang, Y., Zhang, D., Li, J., and Tian, Q. (2017, January 21–26). Task-Driven Dynamic Fusion: Reducing Ambiguity in Video Description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, HI, USA.10.1109/CVPR.2017.662 ↗
35. Li, X., Zhao, B., and Lu, X. (2017, January 19–25). MAM-RNN: Multi-level Attention Model Based RNN for Video Captioning. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, Melbourne, Australia.10.24963/ijcai.2017/307 ↗
36. Zhao, B., Li, X., and Lu, X. (2018, January 13–19). Video Captioning with Tube Features. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, Stockholm, Sweden.10.24963/ijcai.2018/164 ↗
37. Video description with subject, verb and object supervision
38. STAT: Spatial-temporal attention mechanism for video captioning10.1109/TMM.2019.2924576 ↗