Att-BiL-SL: Attention-Based Bi-LSTM and Sequential LSTM for Describing Video in the Textual Formation
Source abstract
Abstract
With the advancement of the technological field, day by day, people from around the world are having easier access to internet abled devices, and as a result, video data is growing rapidly. The increase of portable devices such as various action cameras, mobile cameras, motion cameras, etc., can also be considered for the faster growth of video data. Data from these multiple sources need more maintenance to process for various usages according to the needs. By considering these enormous amounts of video data, it cannot be navigated fully by the end-users. Throughout recent times, many research works have been done to generate descriptions from the images or visual scene recordings to address the mentioned issue. This description generation, also known as video captioning, is more complex than single image captioning. Various advanced neural networks have been used in various studies to perform video captioning. In this paper, we propose an attention-based Bi-LSTM and sequential LSTM (Att-BiL-SL) encoder-decoder model for describing the video in textual format. The model consists of two-layer attention-based bi-LSTM and one-layer sequential LSTM for video captioning. The model also extracts the universal and native temporal features from the video frames for smooth sentence generation from optical frames. This paper includes the word embedding with a soft attention mechanism and a beam search optimization algorithm to generate qualitative results. It is found that the architecture proposed in this paper performs better than various existing state of the art models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Video Analysis and Summarization · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, American International University-Bangladesh, 408/1, Kuratoli, Khilkhet, Dhaka 1229, Bangladesh
Institute IR4.0 (IIR4.0), Universiti Kebangsaan Malaysia, Bangi 43600 UKM, Selangor, Malaysia
Skill Development for Mobile Game and Application Project, ICT Division, E-14/X, ICT Tower, Agargaon, Sher-e-Bangla Nagar, Dhaka 1207, Bangladesh
Head of Learning & Development and Public Relations at Genex Infosys Limited, Adjunct Professor at Southeast University, Nitol Niloy Tower, Nikunja-2, Dhaka 1229, Bangladesh
Department of Electrical and Electronics Engineering, BRAC University, 66 Mohakhali, Dhaka 1212, Bangladesh
References (62)
Show source-provided references
1. Cisco (2017). The Zettabyte Era: Trends and Analysis, Cisco. Available online: https://en.wikipedia.org/wiki/Zettabyte_Era.
2. Efficient CNN based summarization of surveillance videos for resource-constrained devices10.1016/j.patrec.2018.08.003 ↗
3. Video Summarization Using Highlight Detection and Pairwise Deep Ranking Model10.1016/j.procs.2020.03.203 ↗
4. Chu, Y.-W., Lin, K.-Y., Hsu, C.-C., and Ku, L.-W. (2020, September 16). Multi-Step Joint-Modality Attention Network for Scene-Aware Dialogue System. Available online: http://arxiv.org/abs/2001.06206.
5. Huang, J.H., and Worring, M. (2020, January 8–11). Query-controllable video summarization. Proceedings of the 2020 International Conference on Multimedia Retrieval, Dublin, Ireland.10.1145/3372278.3390695 ↗
6. Bridging Text and Video: A Universal Multimodal Transformer for Audio-Visual Scene-Aware Dialog10.1109/TASLP.2021.3065823 ↗
7. A bottom-up summarization algorithm for videos in the wild10.1186/s13634-019-0611-y ↗
8. Learning explicit video attributes from mid-level representation for video captioning10.1016/j.cviu.2017.06.012 ↗
9. Hierarchical attention-based multimodal fusion for video captioning10.1016/j.neucom.2018.07.029 ↗
10. Exploring diverse and fine-grained caption for video by incorporating convolutional architecture into LSTM-based model10.1016/j.patrec.2019.11.003 ↗
11. Recurrent convolutional video captioning with global and local attention10.1016/j.neucom.2019.08.042 ↗
12. Semantic-filtered Soft-Split-Aware video captioning with audio-augmented feature10.1016/j.neucom.2019.05.027 ↗
13. Video Captioning by Adversarial LSTM10.1109/TIP.2018.2855422 ↗
14. From Deterministic to Generative: Multimodal Stochastic RNNs for Video Captioning10.1109/TNNLS.2018.2851077 ↗
15. Video Captioning with Attention-Based LSTM and Semantic Consistency10.1109/TMM.2017.2729019 ↗
16. Bin, Y., Yang, Y., Shen, F., Xu, X., and Shen, H.T. (2016, January 15–19). Bidirectional long-short term memory for video description. Proceedings of the 24th ACM international conference on Multimedia, Amsterdam, The Netherland.10.1145/2964284.2967258 ↗
17. Zhu, Y., and Jiang, S. (2019, January 21–25). Attention-based densely connected LSTM for video captioning. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3350932 ↗
18. Francis, D., and Huet, B. (2019, January 21). L-STAP: Learned spatio-temporal adaptive pooling for video captioning. Proceedings of the 1st International Workshop on AI for Smart TV Content Production, Access and Delivery-AI4TV’19, Nice, France.10.1145/3347449.3357484 ↗
19. Dual-Stream Recurrent Neural Network for Video Captioning10.1109/TCSVT.2018.2867286 ↗
20. CAM-RNN: Co-Attention Model Based RNN for Video Captioning10.1109/TIP.2019.2916757 ↗
21. Video Captioning with Adaptive Attention and Mixed Loss Optimization10.1109/ACCESS.2019.2942000 ↗
22. Stateful human-centered visual captioning system to aid video surveillance10.1016/j.compeleceng.2019.07.009 ↗
23. Shi, X., Cai, J., Joty, S., and Gu, J. (2019, January 21–25). Watch it twice: Video captioning with a refocused video encoder. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.10.1145/3343031.3351060 ↗
24. Gui, Y., Guo, D., and Zhao, Y. (2019, January 25). Semantic enhanced encoder-decoder network (SEN) for video captioning. Proceedings of the 2nd Workshop on Multimedia for Accessible Human Computer Interfaces-MAHCI’19, Nice, France.10.1145/3347319.3356839 ↗
25. Qi, M., Wang, Y., Li, A., and Luo, J. (2018, January 26). Sports video captioning by attentive motion representation based hierarchical recurrent neural networks. Proceedings of the 1st International Workshop on Multimedia Content Analysis in Sports, Seoul, Korea.10.1145/3265845.3265851 ↗
26. Liu, S., Ren, Z., and Yuan, J. (2018, January 22–26). SibNet: Sibling convolutional encoder for video captioning. Proceedings of the 26th ACM international conference on Multimedia, Seoul, Korea.10.1145/3240508.3240667 ↗
27. Xiao, H., and Shi, J. (2018, January 16–18). Video captioning using hierarchical multi-attention model. Proceedings of the 2nd International Conference on Advances in Image Processing-ICAIP ’18, Chengdu, China.10.1145/3239576.3239580 ↗
28. Phan, S., Miyao, Y., and Satoh, S. (2017, January 23–27). MANet: A modal attention network for describing videos. Proceedings of the 25th ACM international conference on Multimedia, Mountain View, CA, USA.10.1145/3123266.3127898 ↗
29. Chen, B.-C., Chen, Y.-Y., and Chen, F. (2017, January 4–7). Video to text summary: Joint video summarization and captioning with recurrent neural networks. Proceedings of the 2017 British Machine Vision Conference, London, UK.10.5244/C.31.118 ↗
30. Sah, S., Kulhare, S., Gray, A., Venugopalan, S., Prud’Hommeaux, E., and Ptucha, R. (2017, January 24–31). Semantic text summarization of long videos. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.115 ↗
31. ASoVS: Abstractive Summarization of Video Sequences10.1109/ACCESS.2019.2902507 ↗
32. Fused GRU with semantic-temporal attention for video captioning10.1016/j.neucom.2018.06.096 ↗
33. Hao, X., Zhou, F., and Li, X. (2020, January 12–14). Scene-Edge GRU for Video Caption. Proceedings of the 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chongqing, China.10.1109/ITNEC48623.2020.9084781 ↗
34. Cherian, A., Wang, J., Hori, C., and Marks, T.K. (2020, January 1–5). Spatio-temporal ranked-attention networks for video captioning. Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass, CO, USA.10.1109/WACV45572.2020.9093291 ↗
35. Xu, J., Mei, T., Yao, T., and Rui, Y. (2016, January 27–30). MSR-VTT: A large video description dataset for bridging video and language. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.571 ↗
36. Deep Visual-Semantic Alignments for Generating Image Descriptions10.1109/TPAMI.2016.2598339 ↗
37. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). BLEU: A method for automatic evaluation of machine translation. Proceedings of the the 40th Annual Meeting on Association for Computational Linguistics (ACL ’02), Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
38. Lavie, A., and Agarwal, A. (2007, January 23). METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. Proceedings of the Second Workshop on Statistical Machine Translation, Prague, Czech Republic. Available online: http://acl.ldc.upenn.edu/W/W05/W05-09.pdf#page=75.10.3115/1626355.1626389 ↗
39. Vedantam, R., Zitnick, C.L., and Parikh, D. (2015, January 7–12). CIDEr: Consensus-based image description evaluation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
40. Lin, C.-Y. (2004, January 25–26). ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), Barcelona, Spain.
41. Chen, D., and Dolan, W. (2011, January 19–24). Collecting highly parallel data for paraphrase evaluation. Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, Portland, OR, USA.
42. Simonyan, K., and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. arXiv.
43. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., and Bengio, Y. (2015). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. arXiv.
44. Fang, H., Gupta, S., Iandola, F., Srivastava, R.K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., and Platt, J.C. (2015, January 7–12). From captions to visual concepts and back. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298754 ↗
45. Guadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R., Darrell, T., and Saenko, K. (2013, January 1–8). Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.337 ↗
46. Park, J., Song, C., and Han, J.-H. (2017, January 24–26). A study of evaluation metrics and datasets for video captioning. Proceedings of the 2017 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS), Okinawa, Japan.10.1109/ICIIBMS.2017.8279760 ↗
47. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
48. Chollet, F. (2017, January 21–26). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
49. Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv.
50. Carreira, J., and Zisserman, A. (2017, January 21–26). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
51. Jiang, H., and Learned-Miller, E. (June, January 30). Face Detection with the Faster R-CNN. Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA.10.1109/FG.2017.82 ↗
52. Hochreiter, S., and Schmidhuber, J.J.U. (2021, January 22). Long Short-Term Memory. Available online: http://www7.informatik.tu-muenchen.de/~hochreithttp://www.idsia.ch/~juergen.
53. Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., and Xu, B. (2016, January 7–12). Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany.10.18653/v1/P16-2034 ↗
54. (2021, April 01). Natural Language Toolkit—NLTK 3.5 Documentation. Available online: https://www.nltk.org/.
55. (2021, April 01). TensorFlow. Available online: https://www.tensorflow.org/.
56. (2021, April 01). Keras Applications. Available online: https://keras.io/api/applications/.
57. ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
58. Krishnamoorthy, N., Malkarnenkar, G., Mooney, R., Saenko, K., and Guadarrama, S. (2013, January 14). Generating natural-language video descriptions using text-mined knowledge. Proceedings of the Workshop on Vision and Natural Language Processing, Atlanta, GA, USA. Available online: http://www. aclweb.org/anthology/W13-1302.
59. Guo, J., Liu, H., Li, X., Xu, D., and Zhang, Y. (2021). An Attention Enhanced Spatial–Temporal Graph Convolutional LSTM Network for Action Recognition in Karate. Appl. Sci., 11.10.3390/app11188641 ↗
60. Effective long short-term memory with fruit fly optimization algorithm for time series forecasting10.1007/s00500-020-04855-2 ↗
61. Effective energy consumption forecasting using empirical wavelet transform and long short-term memory10.1016/j.energy.2021.121756 ↗
62. Macroscopic–Microscopic Attention in LSTM Networks Based on Fusion Features for Gear Remaining Life Prediction10.1109/TIE.2019.2959492 ↗