Trends in Event Understanding and Caption Generation/Reconstruction in Dense Video: A Review
Source abstract
Abstract
Video description generates natural language sentences that describe the subject, verb, and objects of the targeted Video. The video description has been used to help visually impaired people to understand the content. It is also playing an essential role in devolving human-robot interaction. The dense video description is more difficult when compared with simple Video captioning because of the object’s interactions and event overlapping. Deep learning is changing the shape of computer vision (CV) technologies and natural language processing (NLP). There are hundreds of deep learning models, datasets, and evaluations that can improve the gaps in current research. This article filled this gap by evaluating some state-of-the-art approaches, especially focusing on deep learning and machine learning for video caption in a dense environment. In this article, some classic techniques concerning the existing machine learning were reviewed. And provides deep learning models, a detail of benchmark datasets with their respective domains. This paper reviews various evaluation metrics, including Bilingual Evaluation Understudy (BLEU), Metric for Evaluation of Translation with Explicit Ordering (METEOR), Word Mover’s Distance (WMD), and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) with their pros and cons. Finally, this article listed some future directions and proposed work for context enhancement using key scene extraction with object detection in a particular frame. Especially, how to improve the context of video description by analyzing key frames detection through morphological image analysis. Additionally, the paper discusses a novel approach involving sentence reconstruction and context improvement through key frame object detection, which incorporates the fusion of large language models for refining results. The ultimate results arise from enhancing the generated text of the proposed model by improving the predicted text and isolating objects using various keyframes. These keyframes identify dense events occurring in the video sequence.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Human Pose and Action Recognition · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (83)
Show source-provided references
1. Context aware video caption generation with consecutive differentiable neural computer
2. Multi-sentence video captioning using content-oriented beam searching and multi-stage refining algorithm10.1016/j.ipm.2020.102302 ↗
3. MM21 Pre-training for video understanding challenge: Video captioning with pretraining techniques
4. Temporal deformable convolutional encoder-decoder networks for video captioning
5. Transfer learning10.4018/978-1-60566-766-9.ch011 ↗
6. Grounding physical concepts of objects and events through dynamic visual reasoning
7. Video captioning based on both egocentric and exocentric views of robot vision for human-robot interaction10.1007/s12369-021-00842-1 ↗
8. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for MS-COCO
9. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions10.1162/tacl_a_00166 ↗
10. Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning
11. Algorithmic characterization of vehicle trajectories from image sequences by motion verbs
12. The “Inverse Hollywood problem”: From video to scripts and storyboards via causal analysis
13. Autonomous complex knowledge mining and graph representation through natural language processing and transfer learning10.1016/j.autcon.2023.105074 ↗
14. Residual attention-based LSTM for video captioning10.1007/s11280-018-0531-z ↗
15. End-to-end video captioning
16. Collecting highly parallel data for paraphrase evaluation
17. MSR-VTT: A large video description dataset for bridging video and language-microsoft research
18. Jointly localizing and describing events for dense video captioning
19. Y. Wang, S. Wang, J. Tang, N. O’Hare, Y. Chang and B. Li, “Hierarchical attention network for action recognition in videos,” arXiv preprint arXiv:1607.06416, 2016.
20. Syntax-aware action targeting for video captioning
21. Video summarization via semantic attended networks
22. Object relational graph with teacher-recommended learning for video captioning
23. Motion guided spatial attention for video captioning
24. Video description: A survey of methods, datasets, and evaluation metrics10.1145/3355390 ↗
25. CAM-RNN: Co-attention model based RNN for video captioning10.1109/TIP.2019.2916757 ↗
26. MAM-RNN: Multi-level attention model based RNN for video captioning
27. Memory-attended recurrent network for video captioning
28. Event-centric multi-modal fusion method for dense video captioning10.1016/j.neunet.2021.11.017 ↗
29. A multi-layer memory sharing network for video captioning10.1016/j.patcog.2022.109202 ↗
30. Machine learning and deep learning for sentiment analysis across languages: A survey10.1016/j.neucom.2023.02.015 ↗
31. F. Li et al., “Temporal modeling approaches for large-scale youtube-8m video understanding,” arXiv preprint arXiv:1707.04555, 2017.
32. Attention-based densely connected LSTM for video captioning
33. Att-BiL-SL: Attention-based Bi-LSTM and sequential LSTM for describing video in the textual formation
34. Adaptively converting auxiliary attributes and textual embedding for video captioning based on BiLSTM10.1007/s11063-020-10352-2 ↗
35. Hierarchical LSTMs with adaptive attention for visual captioning
36. Deep reinforcement learning that matters
37. Video captioning via hierarchical reinforcement learning
38. Less is more: Picking informative frames for video captioning
39. PETS 2009 Benchmark Data, 2009. Accessed: Dec. 06, 2023. [Online]. Available: http://cs.binghamton.edu~mrldata/pets2009.html
40. UCSD Anomaly Detection Dataset, 2022. Accessed: Nov. 07, 2022. [Online]. Available: http://www.svcl.ucsd.edu/projects/anomaly/dataset.html
41. Monitoring Human Activity-Action Recognition, 2022. Accessed: Dec. 06, 2023. [Online]. Available: http://mha.cs.umn.edu/proj_recognition.shtml
42. Real world anomalous scene detection and classification using multilayer deep neural networks
43. A database for fine grained activity detection of cooking activities
44. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching
45. Coherent multi-sentence video description with variable level of detail
46. VideoBERT: A joint model for video and language representation learning
47. A dataset for telling the stories of social media videos
48. L. Ruan, J. Chen, Y. Song, S. Chen, and Q. Jin, “Team RUC_AIM3 technical report at activityNet 2021: Entities object localization,” arXiv preprint arXiv:2106.06138, 2021.
49. A dataset for movie description
50. A. Torabi, C. Pal, H. Larochelle, and A. Courville, “Using descriptive video services to create a large data source for video annotation research,” arXiv preprint arXiv:1503.01070, 2015.
51. Bidirectional long-short term memory for video description
52. Hollywood in homes: Crowdsourcing data collection for activity understanding
53. Title generation for user generated videos
54. Root-mean-square error (RMSE) or mean absolute error (MAE): When to use them or not10.5194/gmd-15-5481-2022 ↗
55. Bleu: A method for automatic evaluation of machine translation
56. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments
57. From word embeddings to document distances
58. Lin · Text Summarization Branches Out · 2004 · 74
59. SPICE: Semantic propositional image caption evaluation
60. K. Li et al., “VideoChat: Chat-centric video understanding,” arXiv preprint arXiv:2305.06355, 2023.
61. GQA: A new dataset for real-world visual reasoning and compositional question answering
62. A corpus of natural language for visual reasoning
63. Image as a foreign language: BEIT pretraining for vision and vision-language tasks
64. L. H. Li, M. Yatskar, D. Yin, C. J. Hsieh, and K. W. Chang, “VisualBERT: A simple and performant baseline for vision and language,” arXiv preprint arXiv:1908.03557, 2019.
65. Winoground: Probing vision and language models for visio-linguistic compositionality
66. M. Yarom et al., “What you see is what you read? improving text-image alignment evaluation,” arXiv preprint arXiv:2305.10400, 2023.
67. Bitton · Advances in Neural Information Processing Systems (NeurIPS) · 2022 · 1
68. ViLT: Vision-and-language transformer without convolution or region supervision
69. R. Wu et al., “Bongard-OpenWorld: Few-shot reasoning for free-form visual concepts in the real world,” arXiv preprint arXiv:2310.10207, 2023.
70. A simple neural attentive meta-learner
71. Visual spatial reasoning10.1162/tacl_a_00566 ↗
72. VASR: Visual analogies of situation recognition
73. LXMert: Learning cross-modality encoder representations from transformers
74. Video swin transformer
75. Alexa, Siri, Cortana, and more: An introduction to voice assistants10.1080/02763869.2018.1404391 ↗
76. AutoAD II: The sequel–who, when, and what in movie audio description
77. H. Zhang, X. Li, and L. Bing, “Video-LLaMA: An instruction-tuned audio-visual language model for video understanding,” arXiv preprint arXiv:2306.02858, 2023.10.18653/v1/2023.emnlp-demo.49 ↗
78. N. Shvetsova, A. Kukleva, X. Hong, C. Rupprecht, B. Schiele and H. Kuehne, “HowToCaption: Prompting LLMs to transform video annotations at scale,” arXiv preprint arXiv:2310.04900, 2023.10.1007/978-3-031-72992-8_1 ↗
79. Audio-enhanced text-to-video retrieval using text-conditioned feature alignment
80. Yang · Advances in Neural Information Processing Systems 29 (NIPS 2016) · 2016 · 2369
81. C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang and N. Duan, “Visual ChatGPT: Talking, drawing and editing with visual foundation models,” arXiv preprint arXiv:2303.04671, 2023.
82. DeepVideo
83. J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” arXiv preprint arXiv:1804.02767, 2018.