Topic-Oriented Text Features Can Match Visual Deep Models of Video Memorability
Source abstract
Abstract
Not every visual media production is equally retained in memory. Recent studies have shown that the elements of an image, as well as their mutual semantic dependencies, provide a strong clue as to whether a video clip will be recalled on a second viewing or not. We believe that short textual descriptions encapsulate most of these relationships among the elements of a video, and thus they represent a rich yet concise source of information to tackle the problem of media memorability prediction. In this paper, we deepen the study of short captions as a means to convey in natural language the visual semantics of a video. We propose to use vector embeddings from a pretrained SBERT topic detection model with no adaptation as input features to a linear regression model, showing that, from such a representation, simpler algorithms can outperform deep visual models. Our results suggest that text descriptions expressed in natural language might be effective in embodying the visual semantics required to model video memorability.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Video Analysis and Summarization · Visual Attention and Saliency Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Information Processing and Telecommunications Center, E.T.S.I. de Telecomunicación, Universidad Politécnica de Madrid, Avda. Complutense 30, 28040 Madrid, Spain
Funding
Ministerio de Ciencia, Innovación y Universidades
References (31)
Show source-provided references
1. What makes a photograph memorable?10.1109/TPAMI.2013.200 ↗
2. Recognition memory for words, sentences, and pictures10.1016/S0022-5371(67)80067-7 ↗
3. Learning 10000 pictures10.1080/14640747308400340 ↗
4. Isola, P., Xiao, J., Torralba, A., and Oliva, A. (2011, January 20–25). What makes an image memorable?. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA.10.1109/CVPR.2011.5995721 ↗
5. Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K.Q. (2011). Understanding the Intrinsic Memorability of Images. Advances in Neural Information Processing Systems, Curran Associates, Inc.
6. Bylinskii, Z., Goetschalckx, L., Newman, A., and Oliva, A. (2021). Memorability: An image-computable measure of information utility. arXiv.10.1007/978-3-030-81465-6_8 ↗
7. Population response magnitude variation in inferotemporal cortex predicts image memorability10.7554/eLife.47596 ↗
8. The intrinsic memorability of face photographs10.1037/a0033872 ↗
9. Baveye, Y., Cohendet, R., Perreira Da Silva, M., and Le Callet, P. (2016, January 15–19). Deep Learning for Image Memorability Prediction: The Emotional Bias. Proceedings of the ACM Multimedia 2016, Amsterdam, The Netherlands.10.1145/2964284.2967269 ↗
10. Conceptual distinctiveness supports detailed visual long-term memory for real-world objects10.1037/a0019165 ↗
11. Scene Memory Is More Detailed Than You Think: The Role of Categories in Visual Long-Term Memory10.1177/0956797610385359 ↗
12. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 3–5). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
13. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Attention is All you Need. Advances in Neural Information Processing Systems, Curran Associates, Inc.
14. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
15. Reimers, N., and Gurevych, I. (2019, January 3–7). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China.10.18653/v1/D19-1410 ↗
16. Khosla, A., Raju, A.S., Torralba, A., and Oliva, A. (2015, January 7–13). Understanding and Predicting Image Memorability at a Large Scale. Proceedings of the International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.275 ↗
17. Ananthabhotla, I., Ramsay, D.B., and Paradiso, J.A. (2019, January 12–17). HCU400: An Annotated Dataset for Exploring Aural Phenomenology Through Causal Uncertainty. Proceedings of the ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683147 ↗
18. Ramsay, D., Ananthabhotla, I., and Paradiso, J. (2019, January 27–29). The Intrinsic Memorability of Everyday Sounds. Proceedings of the Audio Engineering Society Conference: 2019 AES International Conference on Immersive and Interactive Audio, York, UK.
19. Cohendet, R., Demarty, C.H., Duong, N., and Engilberge, M. (2019, January 27–28). VideoMem: Constructing, Analyzing, Predicting Short-Term and Long-Term Video Memorability. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00262 ↗
20. Newman, A., Fosco, C., Casser, V., Lee, A., McNamara, B., and Oliva, A. (2020). Multimodal Memorability: Modeling Effects of Semantics and Decay on Video Memorability. arXiv.10.1007/978-3-030-58517-4_14 ↗
21. Shekhar, S., Singal, D., Singh, H., Kedia, M., and Shetty, A. (2017, January 22–29). Show and Recall: Learning What Makes Videos Memorable. Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy.10.1109/ICCVW.2017.321 ↗
22. Cohendet, R., Yadati, K., Duong, N.Q.K., and Demarty, C.H. (2018, January 11–14). Annotating, Understanding, and Predicting Long-term Video Memorability. Proceedings of the ICMR’18: 2018 International Conference on Multimedia Retrieval, Yokohama, Japan.10.1145/3206025.3206056 ↗
23. Bowman, S.R., Angeli, G., Potts, C., and Manning, C.D. (2015, January 17–21). A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1075 ↗
24. Williams, A., Nangia, N., and Bowman, S. (2018, January 1–6). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA, USA.10.18653/v1/N18-1101 ↗
25. McInnes, L., Healy, J., and Melville, J. (2020). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv.
26. Pei, J., Tseng, V.S., Cao, L., Motoda, H., and Xu, G. (2013). Density-Based Clustering Based on Hierarchical Density Estimates. Advances in Knowledge Discovery and Data Mining, Springer.
27. Jurafsky, D., and Martin, J.H. (2000). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, Prentice Hall PTR. [1st ed.].
28. ImageNet Large Scale Visual Recognition Challenge10.1007/s11263-015-0816-y ↗
29. Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
30. Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., and Garnett, R. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems, Curran Associates, Inc.
31. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.