Survey of Dense Video Captioning: Techniques, Resources, and Future Perspectives
Source abstract
Abstract
Dense Video Captioning (DVC) represents the cutting edge of advanced multimedia tasks, focusing on generating a series of temporally precise descriptions for events unfolding within a video. In contrast to traditional video captioning, which usually offers a singular summary or caption for an entire video, DVC demands the identification of multiple events within a video, the determination of their exact temporal boundaries, and the production of natural language descriptions for each event. This review paper presents a thorough examination of the latest techniques, datasets, and evaluation protocols in the field of DVC. We categorize and assess existing methodologies, delve into the characteristics, strengths, and limitations of widely utilized datasets, and underscore the challenges and opportunities associated with evaluating DVC models. Furthermore, we pinpoint current research trends, open challenges, and potential avenues for future exploration in this domain. The primary contributions of this review encompass: (1) a comprehensive survey of state-of-the-art DVC techniques, (2) an extensive review of commonly employed datasets, (3) a discussion on evaluation metrics and protocols, and (4) the identification of emerging trends and future directions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Human Pose and Action Recognition · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Technology, Xinjiang Normal University, Urumqi 830054, China
Library of Xinjiang Normal University, Urumqi 830017, China
Funding
National Natural Science Foundation of China · Doctoral Research Foundation of Xinjiang Normal University · Xinjiang Normal University Youth Top Talents Project · Xinjiang Key Research and Development Program
References (100+)
Show source-provided references
1. Video description: A comprehensive survey of deep learning approaches10.1007/s10462-023-10414-6 ↗
2. Krishna, R., Hata, K., Ren, F., Fei-Fei, L., and Niebles, J.C. (2017, January 22–29). Dense-captioning events in videos. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.83 ↗
3. Li, Y., Yao, T., Pan, Y., Chao, H., and Mei, T. (2018, January 18–23). Jointly localizing and describing events for dense video captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition alt, Lake City, UT, USA.10.1109/CVPR.2018.00782 ↗
4. Zhou, L., Xu, C., and Corso, J. (2018, January 2–7). Towards automatic learning of procedures from web instructional videos. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12342 ↗
5. Video description: Datasets & evaluation metrics10.1109/ACCESS.2021.3108565 ↗
6. Hou, R., Chen, C., and Shah, M. (2017, January 22–29). Tube convolutional neural network (t-cnn) for action detection in videos. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.620 ↗
7. Xu, J., Mei, T., Yao, T., and Rui, Y. (2016, January 27–30). Msr-vtt: A large video description dataset for bridging video and language. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.571 ↗
8. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002, January 7–12). Bleu: A method for automatic evaluation of machine translation. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.
9. Lavie, A., and Agarwal, A. (2007, January 23). Meteor: An automatic metric for MT evaluation with high levels of correlation with human judgments. Proceedings of the Second Workshop on Statistical Machine Translation, Prague, Czech Republic.10.3115/1626355.1626389 ↗
10. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015, January 7–12). Cider: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
11. Chin-Yew, L. (2004, January 25–26). Rouge: A package for automatic evaluation of summaries. Proceedings of the Workshop on Text Summarization Branches Out, Barcelona, Spain. Available online: https://aclanthology.org/W04-1013/.
12. Fujita, S., Hirao, T., Kamigaito, H., Okumura, M., and Nagata, M. (2020, January 23–28). SODA: Story oriented dense video captioning evaluation framework. Proceedings of the European Conference on Computer Vision, Virtual, Online, UK.10.1007/978-3-030-58539-6_31 ↗
13. Anderson, P., Fernando, B., Johnson, M., and Gould, S. (2016, January 8–16). Spice: Semantic propositional image caption evaluation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46454-1_24 ↗
14. Kusner, M., Sun, Y., Kolkin, N., and Weinberger, K. (2015, January 6–11). From word embeddings to document distances. Proceedings of the International Conference on Machine Learning, Lille, France. Available online: https://proceedings.mlr.press/v37/kusnerb15.
15. Video description: A survey of methods, datasets, and evaluation metrics10.1145/3355390 ↗
16. Spatial-temporal attention mechanism for video captioning10.1109/TMM.2019.2924576 ↗
17. Wang, T., Zhang, R., Lu, Z., Zheng, F., Cheng, R., and Luo, P. (2021, January 11–17). End-to-end dense video captioning with parallel decoding. Proceedings of the IEEE International Conference on Computer Vision, Online, Canada.10.1109/ICCV48922.2021.00677 ↗
18. Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., and Saenko, K. (2014). Translating videos to natural language using deep recurrent neural networks. arXiv.10.3115/v1/N15-1173 ↗
19. Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., and Courville, A. (2015, January 7–13). Describing videos by exploiting temporal structure. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.512 ↗
20. Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J.R., Marks, T.K., and Sumi, K. (2017, January 22–29). Attention-based multimodal fusion for video description. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.450 ↗
21. Huang, G., Pang, B., Zhu, Z., Rivera, C., and Soricut, R. (2020). Multimodal pretraining for dense video captioning. arXiv.10.18653/v1/2020.aacl-main.48 ↗
22. Dense video captioning with early linguistic information fusion10.1109/TMM.2022.3146005 ↗
23. Yu, H., Wang, J., Huang, Z., Yang, Y., and Xu, W. (2016, January 26–30). Video paragraph captioning using hierarchical recurrent neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.496 ↗
24. Pan, Y., Mei, T., Yao, T., Li, H., and Rui, Y. (2016, January 26–30). Jointly modeling embedding and translation to bridge video and language. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA.10.1109/CVPR.2016.497 ↗
25. Wang, X., Chen, W., Wu, J., Wang, Y.-F., and Wang, W.Y. (2018, January 18–22). Video captioning via hierarchical reinforcement learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00443 ↗
26. Hierarchical representation network with auxiliary tasks for video captioning and video question answering10.1109/TIP.2021.3120867 ↗
27. Wang, B., Ma, L., Zhang, W., Jiang, W., Wang, J., and Liu, W. (2019, January 27–2). Controllable video captioning with POS sequence guidance based on gated fusion network. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea.10.1109/ICCV.2019.00273 ↗
28. Memory-based augmentation network for video captioning10.1109/TMM.2023.3295098 ↗
29. Yang, X., Tang, K., Zhang, H., and Cai, J. (2019, January 16–20). Auto-encoding scene graphs for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01094 ↗
30. Zhou, L., Zhou, Y., Corso, J.J., Socher, R., and Xiong, C. (2018, January 18–22). End-to-end dense video captioning with masked transformer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00911 ↗
31. Yang, A., Nagrani, A., Seo, P.H., Miech, A., Pont-Tuset, J., Laptev, I., Sivic, J., and Schmid, C. (2023, January 18–22). Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01032 ↗
32. Gu, X., Chen, G., Wang, Y., Zhang, L., Luo, T., and Wen, L. (2023, January 18–22). Text with knowledge graph augmented transformer for video captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01816 ↗
33. Center-enhanced video captioning model with multimodal semantic alignment10.1016/j.neunet.2024.106744 ↗
34. Jin, T., Huang, S., Chen, M., Li, Y., and Zhang, Z. (2020). SBAT: Video captioning with sparse boundary-aware transformer. arXiv.10.24963/ijcai.2020/88 ↗
35. Im, H., and Choi, Y.-S. (2022). UAT: Universal attention transformer for video captioning. Sensors, 22.10.3390/s22134817 ↗
36. Ye, H., Li, G., Qi, Y., Wang, S., Huang, Q., and Yang, M.-H. (2022, January 19–24). Hierarchical modular network for video captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01741 ↗
37. Coot: Cooperative hierarchical transformer for video-text representation learning
38. Chen, M., Li, Y., Zhang, Z., and Huang, S. (2018, January 14–16). TVT: Two-view transformer network for video captioning. Proceedings of the Asian Conference on Machine Learning, Beijing, China. Available online: https://proceedings.mlr.press/v95/chen18b.
39. Lei, J., Wang, L., Shen, Y., Yu, D., Berg, T.L., and Bansal, M. (2020). Mart: Memory-augmented recurrent transformer for coherent video paragraph captioning. arXiv.10.18653/v1/2020.acl-main.233 ↗
40. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., and Salakhutdinov, R. (2019). Transformer-xl: Attentive language models beyond a fixed-length context. arXiv.10.18653/v1/P19-1285 ↗
41. Collaborative three-stream transformers for video captioning10.1016/j.cviu.2023.103799 ↗
42. Sun, J., Su, Y., Zhang, H., Cheng, Z., Zeng, Z., Wang, Z., Chen, B., and Yuan, X. (2024). SnapCap: Efficient Snapshot Compressive Video Captioning. arXiv.10.3788/AI.2025.10021 ↗
43. Tu, Y., Zhang, X., Liu, B., and Yan, C. (2017, January 23–27). Video description with spatial-temporal attention. Proceedings of the ACM International Conference on Multimedia, Mountain View, CA, USA.10.1145/3123266.3123354 ↗
44. Video captioning with attention-based LSTM and semantic consistency10.1109/TMM.2017.2729019 ↗
45. Semantic-based temporal attention network for Arabic Video Captioning10.1016/j.nlp.2024.100122 ↗
46. Guo, Z., Gao, L., Song, J., Xu, X., Shao, J., and Shen, H.T. (2016, January 15–19). Attention-based LSTM with semantic consistency for videos captioning. Proceedings of the ACM International Conference on Multimedia, Amsterdam, The Netherlands.10.1145/2964284.2967242 ↗
47. An attention based dual learning approach for video captioning10.1016/j.asoc.2021.108332 ↗
48. Lin, K., Gan, Z., and Wang, L. (2020). Multi-modal feature fusion with feature attention for VATEX captioning challenge 2020. arXiv.
49. Chen, D., and Dolan, W.B. (2011, January 19–24). Collecting highly parallel data for paraphrase evaluation. Proceedings of the Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, USA. Available online: https://aclanthology.org/P11-1020/.
50. Grounding action descriptions in videos10.1162/tacl_a_00207 ↗
51. Rohrbach, A., Rohrbach, M., Tandon, N., and Schiele, B. (2015, January 7–12). A dataset for movie description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298940 ↗
52. Li, Y., Song, Y., Cao, L., Tetreault, J., Goldberg, L., Jaimes, A., and Luo, J. (2016, January 27–30). TGIF: A new dataset and benchmark on animated GIF description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.502 ↗
53. Sigurdsson, G.A., Varol, G., Wang, X., Farhadi, A., Laptev, I., and Gupta, A. (2016, January 8–16). Hollywood in homes: Crowdsourcing data collection for activity understanding. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_31 ↗
54. Zeng, K.-H., Chen, T.-H., Niebles, J.C., and Sun, M. (2016, January 8–16). Title generation for user generated videos. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46475-6_38 ↗
55. Kim, J., Rohrbach, A., Darrell, T., Canny, J., and Akata, Z. (2018, January 8–14). Textual explanations for self-driving vehicles. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01216-8_35 ↗
56. Gella, S., Lewis, M., and Rohrbach, M. (November, January 31). A dataset for telling the stories of social media videos. Proceedings of the Conference on Empirical Methods in Natural Language, Brussels, Belgium.
57. M-VAD names: A dataset for video captioning with naming10.1007/s11042-018-7040-z ↗
58. Wang, X., Wu, J., Chen, J., Li, L., Wang, Y.-F., and Wang, W.Y. (November, January 27). Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea.
59. Lei, J., Yu, L., Berg, T.L., and Bansal, M. (2020, January 23–28). TVR: A large-scale dataset for video-subtitle moment retrieval. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58589-1_27 ↗
60. Xi, Z., Shi, G., Wu, L., Li, X., Yan, J., Wang, L., and Liu, Z. (2024). Knowledge Graph Supported Benchmark and Video Captioning for Basketball. arXiv.
61. Kong, Q., Kawana, Y., Saini, R., Kumar, A., Pan, J., Gu, T., Ozao, Y., Opra, B., Sato, Y., and Kobori, N. (October, January 29). Wts: A pedestrian-centric traffic video dataset for fine-grained spatial-temporal understanding. Proceedings of the European Conference on Computer Vision, Milan, Italy.
62. Yang, T., Jia, J., Wang, B., Cheng, Y., Li, Y., Hao, D., Cao, X., Chen, Q., Li, H., and Jiang, P. (2024, January 28–31). Spatiotemporal Fine-grained Video Description for Short Videos. Proceedings of the ACM International Conference on Multimedia, Melbourne, Australia.10.1145/3664647.3681333 ↗
63. Zhang, G., Ren, J., Gu, J., and Tresp, V. (2023, January 2–6). Multi-event video-text retrieval. Proceedings of the IEEE International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.02021 ↗
64. Shen, X., Li, D., Zhou, J., Qin, Z., He, B., Han, X., Li, A., Dai, Y., Kong, L., and Wang, M. (2023, January 18–22). Fine-grained audible video description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01020 ↗
65. Islam, M.M., Ho, N., Yang, X., Nagarajan, T., Torresani, L., and Bertasius, G. (2024, January 17–21). Video ReCap: Recursive Captioning of Hour-Long Videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01723 ↗
66. Kim, M., Kim, H.B., Moon, J., Choi, J., and Kim, S.T. (2024, January 17–21). Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01318 ↗
67. Ohkawa, T., Yagi, T., Nishimura, T., Furuta, R., Hashimoto, A., Ushiku, Y., and Sato, Y. (2023). Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videos. arXiv.
68. Wu, J., Wang, J., Yang, Z., Gan, Z., Liu, Z., Yuan, J., and Wang, L. (October, January 29). Grit: A generative region-to-text transformer for object understanding. Proceedings of the European Conference on Computer Vision, Milan, Italy.
69. Video event description in scene context10.1016/j.neucom.2012.03.037 ↗
70. Zhou, X., Arnab, A., Sun, C., and Schmid, C. (2023). Dense Video Object Captioning from Disjoint Supervision. arXiv.10.1109/CVPR52733.2024.01727 ↗
71. Xiao, J., Wu, Y., Chen, Y., Wang, S., Wang, Z., and Ma, J. (2023, January 18–22). LSTFE-net: Long short-term feature enhancement network for video small object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01404 ↗
72. Gao, H., Xu, H., Cai, Q.-Z., Wang, R., Yu, F., and Darrell, T. (November, January 27). Disentangling propagation and generation for video prediction. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea.
73. Ramanathan, V., Liang, P., and Fei-Fei, L. (2013, January 1–8). Video event understanding using natural language descriptions. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.117 ↗
74. Whitehead, S., Ji, H., Bansal, M., Chang, S.-F., and Voss, C. (November, January 31). Incorporating background knowledge into video description generation. Proceedings of the Conference on Empirical Methods in Natural Language, Brussels, Belgium.
75. Guadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R., Darrell, T., and Saenko, K. (2013, January 1–8). Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.337 ↗
76. Shvetsova, N., Kukleva, A., Hong, X., Rupprecht, C., Schiele, B., and Kuehne, H. (October, January 29). Howtocaption: Prompting llms to transform video annotations at scale. Proceedings of the European Conference on Computer Vision, Milan, Italy.
77. Fully exploring object relation interaction and hidden state attention for video captioning10.1016/j.patcog.2024.111138 ↗
78. ATMNet: Adaptive Two-Stage Modular Network for Accurate Video Captioning10.1109/TSMC.2024.3524682 ↗
79. Chen, H., Li, J., and Hu, X. (2020). Delving deeper into the decoder for video captioning. ECAI 2020, IOS Press.
80. Tan, G., Liu, D., Wang, M., and Zha, Z.-J. (2020). Learning to discretely compose reasoning module networks for video captioning. arXiv.10.24963/ijcai.2020/104 ↗
81. Adaptive semantic guidance network for video captioning10.1016/j.cviu.2024.104255 ↗
82. CroCaps: A CLIP-assisted cross-domain video captioner10.1016/j.eswa.2024.126296 ↗
83. Pseudo-labeling with keyword refining for few-supervised video captioning10.1016/j.patcog.2024.111176 ↗
84. Global semantic enhancement network for video captioning10.1016/j.patcog.2023.109906 ↗
85. EDS: Exploring deeper into semantics for video captioning10.1016/j.patrec.2024.09.017 ↗
86. Style-aware two-stage learning framework for video captioning10.1016/j.knosys.2024.112258 ↗
87. Multi-scale features with temporal information guidance for video captioning10.1016/j.engappai.2024.109102 ↗
88. CMGNet: Collaborative multi-modal graph network for video captioning10.1016/j.cviu.2023.103864 ↗
89. Shen, Y., Gu, X., Xu, K., Fan, H., Wen, L., and Zhang, L. (2023, January 23–28). Accurate and fast compressed video captioning. Proceedings of the IEEE International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01426 ↗
90. Concept-aware video captioning: Describing videos with effective prior information10.1109/TIP.2023.3307969 ↗
91. Visual commonsense-aware representation network for video captioning10.1109/TNNLS.2023.3323491 ↗
92. Zhong, X., Li, Z., Chen, S., Jiang, K., Chen, C., and Ye, M. (2023, January 7–14). Refined semantic enhancement towards frequency diffusion for video captioning. Proceedings of the AAAI Conference on Artificial Intelligence, Washington DC, USA.10.1609/aaai.v37i3.25484 ↗
93. Perez-Martin, J., Bustos, B., and P’erez, J. (2021, January 3–8). Improving video captioning with temporal composition of a visual-syntactic embedding. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV48630.2021.00308 ↗
94. Enhancing the alignment between target words and corresponding frames for video captioning10.1016/j.patcog.2020.107702 ↗
95. Zheng, Q., Wang, C., and Tao, D. (2020, January 13–19). Syntax-aware action targeting for video captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01311 ↗
96. Hou, J., Wu, X., Zhao, W., Luo, J., and Jia, Y. (November, January 27). Joint syntax representation learning and visual cue translation for video captioning. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea.
97. Diverse video captioning through latent variable expansion10.1016/j.patrec.2022.05.021 ↗
98. Liu, F., Ren, X., Wu, X., Yang, B., Ge, S., Zou, Y., and Sun, X. (2021). O2NA: An object-oriented non-autoregressive approach for controllable video captioning. arXiv.10.18653/v1/2021.findings-acl.24 ↗
99. Pan, B., Cai, H., Huang, D.-A., Lee, K.-H., Gaidon, A., Adeli, E., and Niebles, J.C. (2020, January 14–19). Spatio-temporal graph for video captioning with knowledge distillation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01088 ↗
100. Rethink video retrieval representation for video captioning10.1016/j.patcog.2024.110744 ↗