Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend
Source abstract
Abstract
Deep vision multimodal learning aims at combining deep visual representation learning with other modalities, such as text, sound, and data collected from other sensors. With the fast development of deep learning, vision multimodal learning has gained much interest from the community. This paper reviews the types of architectures used in multimodal learning, including feature extraction, modality aggregation, and multimodal loss functions. Then, we discuss several learning paradigms such as supervised, semi-supervised, self-supervised, and transfer learning. We also introduce several practical challenges such as missing modalities and noisy modalities. Several applications and benchmarks on vision tasks are listed to help researchers gain a deeper understanding of progress in the field. Finally, we indicate that pretraining paradigm, unified multitask framework, missing and noisy modality, and multimodal task diversity could be the future trends and challenges in the deep vision multimodal learning field. Compared with existing surveys, this paper focuses on the most recent works and provides a thorough discussion of methodology, benchmarks, and future trends.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Zhejiang University-University of Illinois at Urbana-Champaign Institute, Zhejiang University, Haining 314400, China
Funding
National Natural Science Foundation of China
References (100+)
Show source-provided references
1. What Makes Multi-modal Learning Better than Single (Provably)
2. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A.Y. (28–2, January 28). Multimodal deep learning. Proceedings of the 28th International Conference on Machine Learning, Washington, DC, USA.
3. Multimodal machine learning: A survey and taxonomy
4. Representation learning: A review and new perspectives10.1109/TPAMI.2013.50 ↗
5. Multimodal intelligence: Representation learning, information fusion, and applications10.1109/JSTSP.2020.2987728 ↗
6. Deep multimodal representation learning: A survey10.1109/ACCESS.2019.2916887 ↗
7. Trends in integration of vision and language research: A survey of tasks, datasets, and methods10.1613/jair.1.11688 ↗
8. Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., and Gao, J. (2021, January 20–25). Vinvl: Revisiting visual representations in vision-language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00553 ↗
9. Attention is all you need
10. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
11. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C. (2021, January 10–17). Vivit: A video vision transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00676 ↗
12. Gong, Y., Chung, Y.A., and Glass, J. (2021). Ast: Audio spectrogram transformer. arXiv.10.21437/Interspeech.2021-698 ↗
13. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
14. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
15. Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. (2021). FLAVA: A Foundational Language And Vision Alignment Model. arXiv.10.1109/CVPR52688.2022.01519 ↗
16. Likhosherstov, V., Arnab, A., Choromanski, K., Lucic, M., Tay, Y., Weller, A., and Dehghani, M. (2021). PolyViT: Co-training Vision Transformers on Images, Videos and Audio. arXiv.
17. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text
18. Lee, S., Yu, Y., Kim, G., Breuel, T., Kautz, J., and Song, Y. (2020). Parameter efficient multimodal transformers for video representation learning. arXiv.
19. Jason, W., Sumit, C., and Antoine, B. (2014). Memory Networks. arXiv.
20. End-to-end memory networks
21. Wang, J., Wang, W., Huang, Y., Wang, L., and Tan, T. (2018, January 18–23). M3: Multimodal memory modelling for video captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00784 ↗
22. Lin, C., Jiang, Y., Cai, J., Qu, L., Haffari, G., and Yuan, Z. (2021). Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation. arXiv.10.1007/978-3-031-20059-5_22 ↗
23. History aware multimodal transformer for vision-and-language navigation
24. Xiong, C., Merity, S., and Socher, R. (2016, January 19–24). Dynamic memory networks for visual and textual question answering. Proceedings of the International Conference on Machine Learning, New York, NY USA.
25. Early, intermediate and late fusion strategies for robust deep learning-based multimodal action recognition10.1007/s00138-021-01249-8 ↗
26. Multisensor data fusion: A review of the state-of-the-art10.1016/j.inffus.2011.08.001 ↗
27. Deep dynamic neural networks for multimodal gesture segmentation and recognition10.1109/TPAMI.2016.2537340 ↗
28. Kahou, S.E., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, Ç., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., and Ferrari, R.C. (2013, January 9–13). Combining modality specific deep neural networks for emotion recognition in video. Proceedings of the 15th ACM on International Conference on Multimodal Interaction, Sydney, Australia.10.1145/2522848.2531745 ↗
29. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014, January 23–28). Large-scale video classification with convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.223 ↗
30. Moddrop: Adaptive multi-modal gesture recognition10.1109/TPAMI.2015.2461544 ↗
31. Ma, M., Ren, J., Zhao, L., Testuggine, D., and Peng, X. (2022, January 3–8). Are Multimodal Transformers Robust to Missing Modality?. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Waikoloa, HI, USA.10.1109/CVPR52688.2022.01764 ↗
32. Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.P. (2017). Tensor fusion network for multimodal sentiment analysis. arXiv.10.18653/v1/D17-1115 ↗
33. Deep multimodal multilinear fusion with high-order polynomial pooling
34. Xu, R., Xiong, C., Chen, W., and Corso, J. (2015, January 19–25). Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. Proceedings of the AAAI Conference on Artificial Intelligence, Austin, TX, USA.10.1609/aaai.v29i1.9512 ↗
35. Sahu, G., and Vechtomova, O. (2019). Dynamic fusion for multimodal data. arXiv.
36. Zadeh, A., Liang, P.P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.P. (2018, January 2–7). Memory fusion network for multi-view sequential learning. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LO, USA.10.1609/aaai.v32i1.12021 ↗
37. Xu, N., Mao, W., and Chen, G. (2019, January 27). Multi-interactive memory network for aspect based multimodal sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.10.1609/aaai.v33i01.3301371 ↗
38. Attention bottlenecks for multimodal fusion
39. Pérez-Rúa, J.M., Vielzeuf, V., Pateux, S., Baccouche, M., and Jurie, F. (2019, January 15–20). Mfas: Multimodal fusion architecture search. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00713 ↗
40. Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., and Morency, L.P. (2018). Efficient low-rank multimodal fusion with modality-specific factors. arXiv.10.18653/v1/P18-1209 ↗
41. Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies
42. George, A., and Marcel, S. (2021, January 20–25). Cross modal focal loss for rgbd face anti-spoofing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00779 ↗
43. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
44. Jing, L., Vahdani, E., Tan, J., and Tian, Y. (2020). Cross-modal center loss. arXiv.
45. Wen, Y., Zhang, K., Li, Z., and Qiao, Y. (2016). A discriminative feature learning approach for deep face recognition. European Conference on Computer Vision, Springer.10.1007/978-3-319-46478-7_31 ↗
46. Coot: Cooperative hierarchical transformer for video-text representation learning
47. Tan, H., and Bansal, M. (2019). Lxmert: Learning cross-modality encoder representations from transformers. arXiv.10.18653/v1/D19-1514 ↗
48. Zhu, L., and Yang, Y. (2020, January 13–19). Actbert: Learning global-local video-text representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00877 ↗
49. Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., and Carion, N. (2021, January 11–17). MDETR-modulated detection for end-to-end multi-modal understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00180 ↗
50. Van den Oord, A., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv.
51. Chun, S., Oh, S.J., De Rezende, R.S., Kalantidis, Y., and Larlus, D. (2021, January 20–25). Probabilistic embeddings for cross-modal retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00831 ↗
52. Hadsell, R., Chopra, S., and LeCun, Y. (2006, January 17–22). Dimensionality reduction by learning an invariant mapping. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), New York, NY, USA.
53. Valverde, F.R., Hurtado, J.V., and Valada, A. (2021, January 20–25). There is more than meets the eye: Self-supervised multi-object detection and tracking with sound by distilling multimodal knowledge. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01144 ↗
54. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.
55. Introduction to semi-supervised learning
56. Guillaumin, M., Verbeek, J., and Schmid, C. (2010, January 13–18). Multimodal semi-supervised learning for image classification. Proceedings of the 2010 IEEE Computer society conference on computer vision and pattern recognition, San Francisco, CA, USA.10.1109/CVPR.2010.5540120 ↗
57. Cheng, Y., Zhao, X., Cai, R., Li, Z., Huang, K., and Rui, Y. (2016, January 9–15). Semi-Supervised Multimodal Deep Learning for RGB-D Object Recognition. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, New York, NY, USA.
58. Cheng, Y., Zhao, X., Huang, K., and Tan, T. (2014, January 24–28). Semi-supervised learning for rgb-d object recognition. Proceedings of the 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden.10.1109/ICPR.2014.412 ↗
59. Tian, D., Gong, M., Zhou, D., Shi, J., and Lei, Y. (2017). Semi-supervised multimodal hashing. arXiv.
60. Shen, Y., Zhang, L., and Shao, L. (June, January 29). Semi-supervised vision-language mapping via variational learning. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989160 ↗
61. Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., and Tang, J. (2021). Self-supervised learning: Generative or contrastive. IEEE Trans. Knowl. Data Eng., Early Access.10.1109/TKDE.2021.3090866 ↗
62. Taleb, A., Lippert, C., Klein, T., and Nabi, M. (2021). Multimodal self-supervised learning for medical image analysis. International Conference on Information Processing in Medical Imaging, Springer.10.1007/978-3-030-78191-0_51 ↗
63. Tamkin, A., Liu, V., Lu, R., Fein, D., Schultz, C., and Goodman, N. (2021). DABS: A Domain-Agnostic Benchmark for Self-Supervised Learning. arXiv.
64. Coen, M.H. (2006). Multimodal Dynamics: Self-Supervised Learning in Perceptual and Motor Systems. [Ph.D. Thesis, Massachusetts Institute of Technology].
65. Gomez, L., Patel, Y., Rusinol, M., Karatzas, D., and Jawahar, C. (2017, January 21–26). Self-supervised learning of visual features through embedding images into text topic spaces. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.218 ↗
66. Imagenet classification with deep convolutional neural networks
67. Latent dirichlet allocation
68. Afouras, T., Owens, A., Chung, J.S., and Zisserman, A. (2020). Self-supervised learning of audio-visual objects from video. European Conference on Computer Vision, Springer.10.1007/978-3-030-58523-5_13 ↗
69. Labelling unlabelled videos from scratch with multi-modal self-supervision
70. Self-supervised multimodal versatile networks
71. Cheng, Y., Wang, R., Pan, Z., Feng, R., and Zhang, Y. (2020, January 12–16). Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.10.1145/3394171.3413869 ↗
72. Self-supervised learning by cross-modal audio-video clustering
73. A survey of transfer learning10.1186/s40537-016-0043-6 ↗
74. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2022, June 01). Improving Language Understanding by Generative Pre-Training. Available online: https://openai.com/blog/language-unsupervised/.
75. Language models are unsupervised multitask learners
76. Language models are few-shot learners
77. Schneider, S., Baevski, A., Collobert, R., and Auli, M. (2019). wav2vec: Unsupervised pre-training for speech recognition. arXiv.10.21437/Interspeech.2019-1873 ↗
78. Yu, W., Xu, H., Yuan, Z., and Wu, J. (2021). Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. arXiv.10.1609/aaai.v35i12.17289 ↗
79. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., and Dai, J. (2019). Vl-bert: Pre-training of generic visual-linguistic representations. arXiv.
80. Hu, R., and Singh, A. (2021, January 10–17). Unit: Multimodal multitask learning with a unified transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00147 ↗
81. Chen, F., Zhang, D., Han, M., Chen, X., Shi, J., Xu, S., and Xu, B. (2022). VLP: A Survey on Vision-Language Pre-training. arXiv.10.1007/s11633-022-1369-5 ↗
82. Li, G., Duan, N., Fang, Y., Gong, M., and Jiang, D. (2020, January 7). Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6795 ↗
83. Zhou, M., Zhou, L., Wang, S., Cheng, Y., Li, L., Yu, Z., and Liu, J. (2021, January 19–25). Uc2: Universal cross-lingual cross-modal vision-and-language pre-training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00414 ↗
84. Integrating multimodal information in large pretrained transformers
85. Wang, Y., Shen, Y., Liu, Z., Liang, P.P., Zadeh, A., and Morency, L.P. (2019, January 27). Words can shift: Dynamically adjusting word representations using nonverbal behaviors. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.10.1609/aaai.v33i01.33017216 ↗
86. Large-scale adversarial training for vision-and-language representation learning
87. Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., and McDaniel, P. (2017). Ensemble adversarial training: Attacks and defenses. arXiv.
88. Xie, S., Sun, C., Huang, J., Tu, Z., and Murphy, K. (2018, January 8–14). Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. Proceedings of the European conference on computer vision (ECCV), München, Germany.10.1007/978-3-030-01267-0_19 ↗
89. Feichtenhofer, C., Fan, H., Malik, J., and He, K. (November, January 27). Slowfast networks for video recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
90. Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T.L., Bansal, M., and Liu, J. (2021, January 20–25). Less is more: Clipbert for video-and-language learning via sparse sampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00725 ↗
91. Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. (November, January 27). Videobert: A joint model for video and language representation learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
92. Tan, H., and Bansal, M. (2020). Vokenization: Improving language understanding with contextualized, visual-grounded supervision. arXiv.10.18653/v1/2020.emnlp-main.162 ↗
93. Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., and Peng, X. (2021). Smil: Multimodal learning with severely missing modality. arXiv.10.1609/aaai.v35i3.16330 ↗
94. Unconstrained multimodal multi-label learning10.1109/TMM.2015.2476658 ↗
95. Ding, Z., Ming, S., and Fu, Y. (2014, January 27). Latent low-rank transfer subspace learning for missing modality recognition. Proceedings of the AAAI Conference on Artificial Intelligence, Quebec City, QC, Canada.10.1609/aaai.v28i1.8905 ↗
96. Missing modality transfer learning via latent low-rank constraint10.1109/TIP.2015.2462023 ↗
97. A survey of multimodal sentiment analysis10.1016/j.imavis.2017.08.003 ↗
98. Pham, H., Liang, P.P., Manzini, T., Morency, L.P., and Póczos, B. (2019, January 27). Found in translation: Learning robust joint representations by cyclic translations between modalities. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.10.1609/aaai.v33i01.33016892 ↗
99. Moon, S., Neves, L., and Carvalho, V. (2018, January 15–20). Multimodal named entity disambiguation for noisy social media posts. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.10.18653/v1/P18-1186 ↗
100. Gupta, T., Schwing, A., and Hoiem, D. (November, January 27). Vico: Word embeddings from visual co-occurrences. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.