Effectively Obtaining Acoustic, Visual, and Textual Data from Videos
Source abstract
Abstract
The increasing use of machine learning models has amplified the demand for high-quality, large-scale multimodal datasets. However, the availability of such datasets, especially those combining acoustic, visual, and textual data, remains limited. This paper addresses this gap by proposing a method of extracting related audio–image–text observations from videos. We detail the process of selecting suitable videos, extracting relevant data pairs, and generating descriptive texts using image-to-text models. Our approach ensures a robust semantic connection between modalities, enhancing the utility of the created datasets for various applications. We also explore the obtained data, discuss the challenges encountered, and propose solutions to improve data quality. The resulting datasets, which are publicly available, aim to support and advance research in multimodal data analysis and machine learning.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Generative Adversarial Networks and Image Synthesis · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Engineering and Science, Adolfo Ibánez University (UAI), Santiago 7941169, Chile
Faculty of Engineering and Science, Diego Portales University (UDP), Santiago 8370191, Chile
References (100+)
Show source-provided references
1. Mukhamediev, R.I., Symagulov, A., Kuchin, Y., Yakunin, K., and Yelis, M. (2021). From Classical Machine Learning to Deep Neural Networks: A Simplified Scientometric Review. Appl. Sci., 11.10.3390/app11125541 ↗
2. Zhang, C., Zhang, C., Zhang, M., and Kweon, I.S. (2023). Text-to-image Diffusion Models in Generative AI: A Survey. arXiv.
3. Franceschelli, G., and Musolesi, M. (2022). Creativity and Machine Learning: A Survey. arXiv.
4. A Survey of On-Device Machine Learning: An Algorithms and Learning Theory Perspective10.1145/3450494 ↗
5. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning Transferable Visual Models from Natural Language Supervision. arXiv.
6. Li, J., Li, D., Xiong, C., and Hoi, S. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. arXiv.
7. Li, J., Li, D., Savarese, S., and Hoi, S. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv.
8. Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. (2023). SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. arXiv.
9. Imagen-Team-Google, Baldridge, J., Bauer, J., Bhutani, M., Brichtova, N., Bunner, A., Castrejon, L., Chan, K., Chen, Y., and Dieleman, S. (2024). Imagen 3. arXiv.
10. Labs, B.F. (2025, November 23). FLUX. Available online: https://github.com/black-forest-labs/flux.
11. Bertalmío, M., Sapiro, G., Caselles, V., and Ballester, C. (2000, January 23–28). Image inpainting. Proceedings of the 27th Internationl Conference on Computer Graphics and Interactive Techniques Conference, New Orleans, LA, USA.10.1145/344779.344972 ↗
12. Outpainting Images and Videos using GANs10.14445/22312803/IJCTT-V68I5P107 ↗
13. Image-to-Image Translation: Methods and Applications10.1109/TMM.2021.3109419 ↗
14. Saxena, S., and Teli, M.N. (2022). Comparison and Analysis of Image-to-Image Generative Adversarial Networks: A Survey. arXiv.
15. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft COCO: Common Objects in Context. Proceedings of the 13th European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
16. Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., and Wortsman, M. (2022, January 6–14). LAION-5B: An open large-scale dataset for training next generation image-text models. Proceedings of the 35th International Conference on Neural Information Processing Systems, online.
17. Changpinyo, S., Sharma, P.K., Ding, N., and Soricut, R. (2021, January 20–25). Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00356 ↗
18. Liu, Z., Luo, P., Qiu, S., Wang, X., and Tang, X. (2016, January 27–30). DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.124 ↗
19. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. arXiv.10.1109/CVPR52688.2022.01042 ↗
20. Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., and Boesel, F. (2024). Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. arXiv.
21. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2025, November 23). Stable Diffusion. Available online: https://github.com/CompVis/stable-diffusion.
22. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-Shot Text-to-Image Generation. arXiv.
23. Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., and Guo, Y. (2025, November 23). Improving Image Generation with Better Captions. Available online: https://cdn.openai.com/papers/dall-e-3.pdf.
24. OpenAI (2025, November 23). DALL·E 3 System Card. Available online: https://cdn.openai.com/papers/DALL_E_3_System_Card.pdf.
25. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., and Gontijo-Lopes, R. (December, January 28). Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA.
26. Birhane, A., and Prabhu, V. (2021, January 3–8). Large image datasets: A pyrrhic win for computer vision?. Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV48630.2021.00158 ↗
27. A survey on Image Data Augmentation for Deep Learning10.1186/s40537-019-0197-0 ↗
28. Wijngaard, G., Formisano, E., Esposito, M., and Dumontier, M. (2024). Audio-Language Datasets of Scenes and Events: A Survey. arXiv.10.1109/ACCESS.2025.3534621 ↗
29. Żelaszczyk, M., and Mańdziuk, J. (2021). Audio-to-Image Cross-Modal Generation. arXiv.
30. Tang, Z., Yang, Z., Khademi, M., Liu, Y., Zhu, C., and Bansal, M. (2023). CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation. arXiv.10.1109/CVPR52733.2024.02589 ↗
31. Tang, Z., Yang, Z., Zhu, C., Zeng, M., and Bansal, M. (2024, January 10–15). Any-to-any generation via composable diffusion. Proceedings of the 37th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
32. Lab, S.A. (2025, November 23). BindDiffusion: One Diffusion Model to Bind Them All. Available online: https://github.com/sail-sg/BindDiffusion.
33. Remote Sensing Image Generation From Audio10.1109/LGRS.2020.2992324 ↗
34. Khanjani, Z., Watson, G., and Janeja, V.P. (2023). Audio deepfakes: A survey. Front. Big Data, 5.10.3389/fdata.2022.1001063 ↗
35. Deep Audio-visual Learning: A Survey10.1007/s11633-021-1293-0 ↗
36. Shi, Z. (2021). A Survey on Audio Synthesis and Audio-Visual Multimodal Processing. arXiv.
37. Sheffer, R., and Adi, Y. (2023). I Hear Your True Colors: Image Guided Audio Generation. arXiv.10.1109/ICASSP49357.2023.10096023 ↗
38. Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., and Li, J. (2023). Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. arXiv.
39. Wu, H., Chen, X., Lin, Y.C., Chang, K.-w., Chung, H.L., Liu, A.H., and Li, H.-y. (2024). Towards audio language modeling—An overview. arXiv.
40. Melechovsky, J., Guo, Z., Ghosal, D., Majumder, N., Herremans, D., and Poria, S. (2024, January 16–21). Mustango: Toward Controllable Text-to-Music Generation. Proceedings of the 2024 North American Chapter of the Association for Computational Linguistics, Mexico City, Mexico.10.18653/v1/2024.naacl-long.459 ↗
41. Dhariwal, P., Jun, H., Payne, C., Kim, J.W., Radford, A., and Sutskever, I. (2020). Jukebox: A Generative Model for Music. arXiv.
42. Valle, R., Badlani, R., Kong, Z., Lee, S.-g., Goel, A., Kim, S., Santos, J.F., Dai, S., Gururani, S., and AIJa’fari, A. (2025, November 23). Fugatto 1—Foundational Generative Audio Transformer Opus 1. Available online: https://openreview.net/forum?id=B2Fqu7Y2cd.
43. Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. (2023). AudioGen: Textually Guided Audio Generation. arXiv.
44. Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M.D. (2023, January 23–29). AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. Proceedings of the 40th International Conference on Machine Learning, Honolulu, HI, USA.
45. Zhan, F., Yu, Y., Wu, R., Zhang, J., Lu, S., Liu, L., Kortylewski, A., Theobalt, C., and Xing, E. (2023). Multimodal Image Synthesis and Editing: The Generative AI Era. arXiv.10.1109/TPAMI.2023.3305243 ↗
46. Multimodal Learning with Transformers: A Survey10.1109/TPAMI.2023.3275156 ↗
47. Transcripter-Generation of the transcript from audio to text using Deep Learning
48. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023, January 23–29). Robust Speech Recognition via Large-Scale Weak Supervision. Proceedings of the 40th International Conference on Machine Learning, Honolulu, HI, USA.
49. Bai, J., Liu, H., Wang, M., Shi, D., Plumbley, M., Gan, W.S., and Chen, J. (2024, January 9–15). AudioSetCaps: An Enriched Audio-Caption Dataset using Automated Generation Pipeline with Large Audio and Language Models. Proceedings of the Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, Vancouver, BC, Canada.
50. Lanzendörfer, L.A., Pinkl, C., Perraudin, N., and Wattenhofer, R. (2024, January 9–15). BLAP: Bootstrapping Language-Audio Pre-training for Music Captioning. Proceedings of the Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, Vancouver, BC, Canada.
51. Xu, X., Zhang, Z., Zhou, Z., Zhang, P., Xie, Z., Wu, M., and Zhu, K.Q. (November, January 29). BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
52. Yariv, G., Gat, I., Wolf, L., Adi, Y., and Schwartz, I. (2023). AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation. arXiv.10.21437/Interspeech.2023-852 ↗
53. Jonason, N., and Sturm, B.L.T. (2022). TimbreCLIP: Connecting Timbre to Text and Images. arXiv.
54. Liu, Y., Zhang, K., Li, Y., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H., and Gao, J. (2024). Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models. arXiv.
55. OpenAI (2025, November 23). Video Generation Models as World Simulators. Available online: https://openai.com/index/video-generation-models-as-world-simulators/.
56. DeepMind, G. (2025, November 23). Veo. Available online: https://deepmind.google/technologies/veo/.
57. Runway (2025, November 23). Introducing Gen-3 Alpha: A New Frontier for Video Generation. Available online: https://runwayml.com/research/introducing-gen-3-alpha.
58. The Movie Gen Team (2025, November 23). Movie Gen: A Cast of Media Foundation Models. Available online: https://ai.meta.com/static-resource/movie-gen-research-paper.
59. A Survey on Generative Diffusion Models10.1109/TKDE.2024.3361474 ↗
60. A survey of multimodal deep generative models10.1080/01691864.2022.2035253 ↗
61. Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.S. (2024). NExT-GPT: Any-to-Any Multimodal LLM. arXiv.
62. Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., and Misra, I. (2023). ImageBind: One Embedding Space To Bind Them All. arXiv.10.1109/CVPR52729.2023.01457 ↗
63. Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. (2021). LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. arXiv.
64. Gemmeke, J.F., Ellis, D.P.W., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., and Ritter, M. (2017, January 5–9). Audio Set: An ontology and human-labeled dataset for audio events. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, New Orleans, LA, USA.10.1109/ICASSP.2017.7952261 ↗
65. Kim, C.D., Kim, B., Lee, H., and Kim, G. (2019, January 3–5). AudioCaps: Generating Captions for Audios in The Wild. Proceedings of the 2019 North American Chapter of the Association for Computational Linguistics, New Minneapolis, MN, USA.
66. Aytar, Y., Vondrick, C., and Torralba, A. (2016, January 20–23). SoundNet: Learning Sound Representations from Unlabeled Video. Proceedings of the 30th International Conference on Neural Information Processing Systems, Changsha, China.10.1109/CVPR.2016.18 ↗
67. Bain, M., Nagrani, A., Varol, G., and Zisserman, A. (2021, January 10–17). Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval. Proceedings of the 2021 IEEE International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00175 ↗
68. Xue, H., Hang, T., Zeng, Y., Sun, Y., Liu, B., Yang, H., Fu, J., and Guo, B. (2022, January 18–24). Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions. Proceedings of the 2022 IEEE International Conference on Computer Vision, New Orleans, LA, USA.10.1109/CVPR52688.2022.00498 ↗
69. Song, S., Lichtenberg, S.P., and Xiao, J. (2015, January 7–12). SUN RGB-D: A RGB-D scene understanding benchmark suite. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298655 ↗
70. Jia, X., Zhu, C., Li, M., Tang, W., and Zhou, W. (2021, January 11–17). LLVIP: A Visible-infrared Paired Dataset for Low-light Vision. Proceedings of the 2021 IEEE International Conference on Computer Vision Workshops, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00389 ↗
71. Grauman, K., Westbury, A., Byrne, E., Cartillier, V., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., and Kukreja, D. (2024). Ego4D: Around the World in 3,000 Hours of Egocentric Video. IEEE Trans. Pattern Anal. Mach. Intell., 1–32.
72. Bie, F., Yang, Y., Zhou, Z., Ghanem, A., Zhang, M., Yao, Z., Wu, X., Holmes, C., Golnari, P., and Clifton, D.A. (2023). RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model. arXiv.
73. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., and Nabeshima, N. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv.
74. Image Generation: A Review10.1007/s11063-022-10777-x ↗
75. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., baptiste Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., and Schrittwieser, J. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv.
76. Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.C., Liu, Z., and Wang, L. (2023). The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision). arXiv.
77. Agostinelli, A., Denk, T.I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., and Tagliasacchi, M. (2023). MusicLM: Generating Music From Text. arXiv.
78. Hong, J., Park, S., and Ro, Y. (2023, January 6–10). Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.10.18653/v1/2023.findings-emnlp.324 ↗
79. Trigka, M., and Dritsas, E. (2025). A Comprehensive Survey of Deep Learning Approaches in Image Processing. Sensors, 25.10.3390/s25020531 ↗
80. Sinha, R.K., Pandey, R., and Pattnaik, R. (2018). Deep Learning For Computer Vision Tasks: A review. arXiv.
81. Deep learning for visual understanding: A review10.1016/j.neucom.2015.09.116 ↗
82. Deep Learning for Computer Vision: A Brief Review10.1155/2018/7068349 ↗
83. A Survey of the Usages of Deep Learning for Natural Language Processing10.1109/TNNLS.2020.2979670 ↗
84. Bai, Y., and Wang, D.Z. (2022). More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering. arXiv.
85. A comprehensive survey of text classification techniques and their research applications: Observational and experimental insights10.1016/j.cosrev.2024.100664 ↗
86. Machine learning in acoustics: Theory and applications10.1121/1.5133944 ↗
87. Deep neural networks for automatic speech processing: A survey from large corpora to limited data10.1186/s13636-022-00251-w ↗
88. Chen, X., Chang, L., Yu, X., Huang, Y., and Tu, X. (2025). A Survey on World Models Grounded in Acoustic Physical Information. arXiv.
89. Audio Describing Sound—What Sounds are Described and How?: Results from a Flemish case study
90. Wu, H.H., Nieto, O., Bello, J.P., and Salamon, J. (2023, January 4–10). Audio-Text Models Do Not Yet Leverage Natural Language. Proceedings of the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, Rhodos, Greece.10.1109/ICASSP49357.2023.10097117 ↗
91. Maks-s (2025, November 23). Stable Diffusion Akashic Records. Available online: https://github.com/Maks-s/sd-akashic.
92. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., and Fan, A. (2024). The Llama 3 Herd of Models. arXiv.
93. Gu, A., and Dao, T. (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv.
94. Mistral AI (2025, November 23). Mistral Models. Available online: https://mistral.ai/technology/#models.
95. Anthropic (2025, November 23). The Claude 3 Model Family: Opus, Sonnet, Haiku. Available online: https://anthropic.com/claude-3-model-card.
96. Koo, R., Lee, M., Raheja, V., Park, J.I., Kim, Z.M., and Kang, D. (2023). Benchmarking Cognitive Biases in Large Language Models as Evaluators. arXiv.10.18653/v1/2024.findings-acl.29 ↗
97. What is noise?10.1190/1.1444411 ↗
98. Huang, R., Long, Y., Han, J., Xu, H., Liang, X., Xu, C., and Liang, X. (2023, January 7–14). NLIP: Noise-Robust Language-Image Pre-training. Proceedings of the 37th AAAI Conference on Artificial Intelligence, Washington, DC, USA.10.1609/aaai.v37i1.25172 ↗
99. Yang, S., Xu, Z., Wang, K., You, Y., Yao, H., Liu, T., and Xu, M. (2023, January 17–24). BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistency. Proceedings of the 2023 IEEE Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01904 ↗
100. Kang, W., Mun, J., Lee, S., and Roh, B. (2023, January 1–6). Noise-Aware Learning from Web-Crawled Image-Text Data for Image Captioning. Proceedings of the 2023 IEEE International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00275 ↗