Multiple Visual-Semantic Embedding for Video Retrieval from Query Sentence
Source abstract
Abstract
Visual-semantic embedding aims to learn a joint embedding space where related video and sentence instances are located close to each other. Most existing methods put instances in a single embedding space. However, they struggle to embed instances due to the difficulty of matching visual dynamics in videos to textual features in sentences. A single space is not enough to accommodate various videos and sentences. In this paper, we propose a novel framework that maps instances into multiple individual embedding spaces so that we can capture multiple relationships between instances, leading to compelling video retrieval. We propose to produce a final similarity between instances by fusing similarities measured in each embedding space using a weighted sum strategy. We determine the weights according to a sentence. Therefore, we can flexibly emphasize an embedding space. We conducted sentence-to-video retrieval experiments on a benchmark dataset. The proposed method achieved superior performance, and the results are competitive to state-of-the-art methods. These experimental results demonstrated the effectiveness of the proposed multiple embedding approach compared to existing methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Human Pose and Action Recognition · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Engineering, Tohoku University, Sendai 9808579, Japan
Funding
Japan Society for the Promotion of Science · Ministry of Education, Culture, Sports, Science and Technology
References (56)
Show source-provided references
1. Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M.A., and Mikolov, T. (2013, January 5–10). DeViSE: A Deep Visual-Semantic Embedding Model. Proceedings of the Advances in Neural Information Processing Systems, Stateline, NV, USA.
2. Engilberge, M., Chevallier, L., Perez, P., and Cord, M. (2018, January 18–22). Finding Beans in Burgers: Deep Semantic-Visual Embedding with Localization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00419 ↗
3. Otani, M., Nakashima, Y., Rahtu, E., Heikkilä, J., and Yokoya, N. (2016). Learning Joint Representations of Videos and Sentences with Web Image Search. European Conference on Computer Vision (ECCV) Workshops, Springer International Publishing.10.1007/978-3-319-46604-0_46 ↗
4. Hendricks, L.A., Wang, O., Shechtman, E., Sivic, J., Darrell, T., and Russell, B. (2017, January 22–29). Localizing Moments in Video with Natural Language. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.618 ↗
5. Gao, J., Sun, C., Yang, Z., and Nevatia, R. (2017, January 22–29). TALL: Temporal Activity Localization via Language Query. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.563 ↗
6. Xu, H., He, K., Plummer, B.A., Sigal, L., Sclaroff, S., and Saenko, K. (2019). Multilevel Language and Vision Integration for Text-to-Clip Retrieval, AAAI.10.1609/aaai.v33i01.33019062 ↗
7. Verma, Y., and Jawahar, C.V. (2012). Image Annotation Using Metric Learning in Semantic Neighbourhoods. Computer Vision—ECCV 2012, Springer.10.1007/978-3-642-33712-3_60 ↗
8. Non-Linear Matrix Completion for Social Image Tagging10.1109/ACCESS.2016.2624267 ↗
9. A Locality Sensitive Low-Rank Model for Image Tag Completion10.1109/TMM.2016.2518478 ↗
10. Deep0Tag: Deep Multiple Instance Learning for Zero-Shot Image Tagging10.1109/TMM.2019.2924511 ↗
11. Deep Visual-Semantic Alignments for Generating Image Descriptions10.1109/TPAMI.2016.2598339 ↗
12. A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their Semantics10.1007/s11263-013-0658-4 ↗
13. Xu, K., Ba, J.L., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R.S., and Bengio, Y. (2015, January 6–11). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Proceedings of the 32nd International Conference on International Conference on Machine Learning, Sydney, Australia.
14. Chen, S., Jin, Q., Wang, P., and Wu, Q. (2020, January 14–19). Say As You Wish: Fine-Grained Control of Image Caption Generation With Abstract Scene Graphs. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00998 ↗
15. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., and Parikh, D. (2015, January 11–18). VQA: Visual Question Answering. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV ’15, Santiago, Chile.10.1109/ICCV.2015.279 ↗
16. Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., and Parikh, D. (July, January 26). Yin and Yang: Balancing and Answering Binary Visual Questions. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.542 ↗
17. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. (2017, January 21–26). Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.670 ↗
18. Jang, Y., Song, Y., Yu, Y., Kim, Y., and Kim, G. (2017, January 21–26). TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.149 ↗
19. Hossain, M.Z., Sohel, F., Shiratuddin, M.F., and Laga, H. (2019). A Comprehensive Survey of Deep Learning for Image Captioning. ACM Comput. Surv., 51.10.1145/3295748 ↗
20. Wang, K., Yin, Q., Wang, W., Wu, S., and Wang, L. (2016). A Comprehensive Survey on Cross-modal Retrieval. arXiv.
21. Ji, Z., Wang, H., Han, J., and Pang, Y. (November, January 27). Saliency-Guided Attention Network for Image-Sentence Matching. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.
22. Wu, H., Mao, J., Zhang, Y., Jiang, Y., Li, L., Sun, W., and Ma, W. (2019, January 16–20). Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00677 ↗
23. Gu, J., Cai, J., Joty, S., Niu, L., and Wang, G. (2018, January 18–22). Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00750 ↗
24. Image and Sentence Matching via Semantic Concepts and Order Learning10.1109/TPAMI.2018.2883466 ↗
25. Kiros, R., Salakhutdinov, R., and Zemel, R.S. (2014). Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models. arXiv.
26. Wang, L., Li, Y., and Lazebnik, S. (July, January 26). Learning Deep Structure-Preserving Image-Text Embeddings. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
27. Ye, K., and Kovashka, A. (2018, January 8–14). ADVISE: Symbolism and External Knowledge for Decoding Advertisements. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01267-0_51 ↗
28. Faghri, F., Fleet, D.J., Kiros, J.R., and Fidler, S. (2018, January 3–6). VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. Proceedings of the British Machine Vision Conference (BMVC), Newcastle, UK.
29. Zhang, D., Dai, X., Wang, X., Wang, Y., and Davis, L.S. (2019, January 16–20). MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00134 ↗
30. Tsai, Y.H., Divvala, S., Morency, L., Salakhutdinov, R., and Farhadi, A. (2019, January 16–20). Video Relationship Reasoning Using Gated Spatio-Temporal Energy Graph. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01067 ↗
31. Song, Y., and Soleymani, M. (2019, January 16–20). Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00208 ↗
32. Mithun, N.C., Li, J., Metze, F., and Roy-Chowdhury, A.K. (2018, January 11–14). Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval. Proceedings of the ACM on International Conference on Multimedia Retrieval, Yokohama, Japan.10.1145/3206025.3206064 ↗
33. Pennington, J., Socher, R., and Manning, C. (2014, January 25–29). Glove: Global Vectors for Word Representation. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
34. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014, January 8–13). Empirical evaluation of gated recurrent neural networks on sequence modeling. Proceedings of the NIPS 2014 Workshop on Deep Learning, Montreal, QC, Canada.
35. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
36. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
37. Adaptive Mixtures of Local Experts10.1162/neco.1991.3.1.79 ↗
38. Kiros, R., Salakhutdinov, R., and Zemel, R. (2014, January 21–26). Multimodal Neural Language Models. Proceedings of the 31st International Conference on Machine Learning, Proceedings of Machine Learning Research, PMLR, Beijing, China.
39. Large Scale Online Learning of Image Similarity Through Ranking
40. Frome, A., Singer, Y., Sha, F., and Malik, J. (2007, January 14–20). Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification. Proceedings of the IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil.10.1109/ICCV.2007.4408839 ↗
41. Grounded Compositional Semantics for Finding and Describing Images with Sentences10.1162/tacl_a_00177 ↗
42. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015, January 11–18). Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.11 ↗
43. Carreira, J., and Zisserman, A. (2017, January 21–26). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
44. Qiu, Z., Yao, T., and Mei, T. (2017, January 22–29). Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.590 ↗
45. Wang, X., Girshick, R., Gupta, A., and He, K. (2018, January 18–22). Non-local Neural Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00813 ↗
46. Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., and Van Gool, L. (2016, January 8–16). Temporal Segment Networks: Towards Good Practices for Deep Action Recognition. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_2 ↗
47. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 11–18). Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.510 ↗
48. Feichtenhofer, C., Fan, H., Malik, J., and He, K. (November, January 27). SlowFast Networks for Video Recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
49. Zhang, S., Guo, S., Huang, W., Scott, M.R., and Wang, L. (May, January 26). V4D: 4D Convolutional Neural Networks for Video-level Representation Learning. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
50. Simonyan, K., and Zisserman, A. (2014, January 8–13). Two-Stream Convolutional Networks for Action Recognition in Videos. Proceedings of the International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
51. Laptev, I., Marszalek, M., Schmid, C., and Rozenfeld, B. (2008, January 23–28). Learning realistic human actions from movies. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.10.1109/CVPR.2008.4587756 ↗
52. Dalal, N., Triggs, B., and Schmid, C. (2006, January 7–13). Human Detection Using Oriented Histograms of Flow and Appearance. Proceedings of the European Conference on Computer Vision, Graz, Austria.10.1007/11744047_33 ↗
53. Wang, H., and Schmid, C. (2013, January 1–8). Action Recognition with Improved Trajectories. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.441 ↗
54. Xu, J., Mei, T., Yao, T., and Rui, Y. (July, January 26). MSR-VTT: A Large Video Description Dataset for Bridging Video and Language. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
55. Dong, J., Li, X., Xu, C., Ji, S., He, Y., Yang, G., and Wang, X. (2019, January 16–20). Dual Encoding for Zero-Example Video Retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00957 ↗
56. Shrivastava, A., Gupta, A., and Girshick, R. (July, January 26). Training Region-Based Object Detectors with Online Hard Example Mining. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.