A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications
Source abstract
Abstract
Multi-modal fusion technology gradually become a fundamental task in many fields, such as autonomous driving, smart healthcare, sentiment analysis, and human-computer interaction. It is rapidly becoming the dominant research due to its powerful perception and judgment capabilities. Under complex scenes, multi-modal fusion technology utilizes the complementary characteristics of multiple data streams to fuse different data types and achieve more accurate predictions. However, achieving outstanding performance is challenging because of equipment performance limitations, missing information, and data noise. This paper comprehensively reviews existing methods based on multi-modal fusion techniques and completes a detailed and in-depth analysis. According to the data fusion stage, multi-modal fusion has four primary methods: early fusion, deep fusion, late fusion, and hybrid fusion. The paper surveys the three major multi-modal fusion technologies that can significantly enhance the effect of data fusion and further explore the applications of multi-modal fusion technology in various fields. Finally, it discusses the challenges and explores potential research opportunities. Multi-modal tasks still need intensive study because of data heterogeneity and quality. Preserving complementary information and eliminating redundant information between modalities is critical in multi-modal technology. Invalid data fusion methods may introduce extra noise and lead to worse results. This paper provides a comprehensive and detailed summary in response to these challenges.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Video Surveillance and Tracking Methods · Anomaly Detection Techniques and Applications · Advanced Chemical Sensor Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (100+)
Show source-provided references
1. Analysis of multimodal data fusion from an information theory perspective10.1016/j.ins.2022.12.014 ↗
2. Effective techniques for multi-modal data fusion: A comparative analysis10.3390/s23052381 ↗
3. Privacy and integrity protection for IoT multimodal data using machine learning and blockchain10.1145/3696453 ↗
4. Multimodal human computer interaction using hand gestures and speech
5. Multi-modality sensing and data fusion for multi-vehicle detection10.1109/TMM.2022.3145663 ↗
6. Multi-modal medical image fusion towards future research: A review
7. Integration of acoustic and visual speech signals using neural networks10.1109/35.41402 ↗
8. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions
9. Multimodal machine learning: A survey and taxonomy10.1109/TPAMI.2018.2798607 ↗
10. Deep multi-modal fusion for semantic image segmentation: A survey10.1016/j.imavis.2020.104042 ↗
11. PointPainting: Sequential fusion for 3D object detection
12. Multi-modal 3D object detection in autonomous driving: A survey10.1007/s11263-023-01784-z ↗
13. Attention is all you need
14. ObjectFusion: Multi-modal 3D object detection with object-centric fusion
15. Virtual sparse convolution for multimodal 3D object detection
16. Continual detection transformer for incremental object detection
17. DynStatF: An efficient feature fusion strategy for LiDAR 3D object detection
18. Are we ready for autonomous driving? The kitti vision bench-mark suite
19. Scalability in perception for autonomous driving: Waymo open dataset
20. nuScenes: A multi-modal dataset for autonomous driving
21. The apolloscape dataset for autonomous driving10.1109/TPAMI.2019.2926463 ↗
22. Pham · May 2020 · 2267
23. PandaSet: Advanced sensor suite dataset for autonomous driving
24. Cirrus: A long-range bi-pattern lidar dataset
25. The H3D dataset for full-surround 3D multi-object detection and tracking in crowded urban scenes
26. Argoverse: 3D tracking and forecasting with rich maps
27. One million scenes for autonomous driving: Once dataset
28. DEAP: A database for emotion analysis; using physiological signals10.1109/T-AFFC.2011.15 ↗
29. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks10.1109/TAMD.2015.2431497 ↗
30. The multi-modal brain tumor image segmentation benchmark (BRATS)10.1109/TMI.2014.2377694 ↗
31. Review the cancer genome atlas (TCGA): An immeasurable source of knowledge
32. C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, “Towards generalist foundation model for radiology,” arXiv preprint arXiv:2308.02463, 2023.
33. C. Wu, X. Zhang, Y. Wang, Y. Zhang, and W. Xie, “K-Diag: Knowledge-enhanced disease diagnosis in radiographic imaging,” arXiv preprint arXiv:2302.11557, 2023.
34. X. Zhang et al., “PMC-VQA: Visual instruction tuning for medical visual question answering,” arXiv preprint arXiv:2305.10415, 2023.
35. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering
36. VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography10.1038/s41597-023-02100-7 ↗
37. A review of natural language processing techniques for opinion mining systems10.1016/j.inffus.2016.10.004 ↗
38. A comprehensive survey on deep learning-based approaches for multimodal sentiment analysis10.1007/s10462-023-10555-8 ↗
39. Multi-modal analysis and prediction of persuasiveness in online social multimedia10.1145/2897739 ↗
40. Multi-modal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph
41. CH-SIMS: A Chinese multi-modal sentiment analysis dataset with fine-grained annotation of modality
42. Towards multi-modal sentiment analysis: Harvesting opinions from the web
43. A. Zadeh, R. Zellers, E. Pincus, and L. P. Morency, “MOSI: Multi-modal corpus of sentiment intensity and subjectivity analysis in online opinion videos,” arXiv preprint arXiv:1606.06259, 2016.
44. The multi-modal sentiment analysis in car reviews (MuSe-CaR) dataset: Collection, insights and improvements10.1109/TAFFC.2021.3097002 ↗
45. S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “MELD: A multi-modal multi-party dataset for emotion recognition in conversations,” arXiv preprint arXiv:1810.02508, 2018.10.18653/v1/P19-1050 ↗
46. Memotion 2: Dataset on sentiment and emotion analysis of memes
47. FACTIFY: A multi-modal fact verification dataset
48. Introducing wesad, a multi-modal dataset for wearable stress and affect detection
49. Representation learning: A review and new perspectives10.1109/TPAMI.2013.50 ↗
50. Multi-modal knowledge representation: A survey
51. Graph to grid: Learning deep representations for multimodal emotion recognition
52. MMHFNet: Multi-modal and multi-layer hybrid fusion network for voice pathology detection10.1016/j.eswa.2023.119790 ↗
53. Multi-modal person re-identification based on transformer relational regularization10.1016/j.inffus.2023.102128 ↗
54. Joint representation learning for text and 3D point cloud10.1016/j.patcog.2023.110086 ↗
55. WiVi-GR: Wireless-visual joint representation-based accurate gesture recognition10.1109/JIOT.2023.3292376 ↗
56. Image-text matching using multi-subspace joint representation10.1007/s00530-022-01038-x ↗
57. Deep multimodal representation learning: A survey10.1109/ACCESS.2019.2916887 ↗
58. RGB-IR person re-identification by cross-modality similarity preservation10.1007/s11263-019-01290-1 ↗
59. Correlation-guided discriminative cross-modality features network for infrared and visible image fusion
60. CoDF‐Net: Coordinated‐representation decision fusion network for emotion recognition with EEG and eye movement signals10.1007/s13042-023-01964-w ↗
61. UWAT-GAN: Fundus fluorescein angiography synthesis via ultra-wide-angle transformation multi-scale GAN
62. Natural language description of human activities from video images based on concept hierarchy of actions10.1023/A:1020346032608 ↗
63. Midge: Generating image descriptions from computer vision detections
64. Image description using visual dependency representations
65. Translating videos to natural language using deep recurrent neural networks
66. The long-short story of movie description
67. HS2P: Hierarchical spectral and structure-preserving usion network for multimodal remote sensing image cloud and shadow removal10.1016/j.inffus.2023.02.002 ↗
68. AV-TranSpeech: Audio-visual robust speech-to-speech translation
69. METTS: Multilingual emotional text-to-speech by cross-speaker and cross-lingual emotion transfer10.1109/TASLP.2024.3363444 ↗
70. Generating question relevant captions to aid visual question answering
71. Going beyond nouns with vision & language models using synthetic data
72. Disentanglement translation network for multimodal sentiment analysis10.1016/j.inffus.2023.102031 ↗
73. TEDT: Transformer-based encoding-decoding translation network for multimodal sentiment analysis10.1007/s12559-022-10073-9 ↗
74. CAMEL: Capturing metaphorical alignment with context disentangling for multimodal emotion recognition
75. W. Elisa et al., “Multi-modal machine learning in image-based and clinical biomedicine: Survey and prospects,” arXiv preprint arXiv:2311.02332, 2023.
76. Answer-based entity extraction and alignment for visual text question answering
77. Probing the impacts of visual context in multimodal entity alignment10.1007/s41019-023-00208-9 ↗
78. Adversarial modality alignment network for cross-modal molecule retrieval10.1109/TAI.2023.3254518 ↗
79. Structural regression fusion for unsupervised multimodal change detection
80. EAF-WGAN: Enhanced alignment fusion-wasserstein generative adversarial network for turbulent image restoration10.1109/TCSVT.2023.3262685 ↗
81. Multi-modal domain adaptation for fine-grained action recognition
82. VideoBERT: A joint model for video and language representation learning
83. Representation, alignment, fusion: A generic transformer-based framework for multi-modal glaucoma recognition
84. AutoAlign: Pixel-instance feature aggregation for multi-modal 3D object detection
85. DeepFusion: LiDAR-camera deep fusion for multi-modal 3D object detection
86. Deep learning for image and point cloud fusion in autonomous driving: A review10.1109/TITS.2020.3023541 ↗
87. Deep learning for lidar points clouds in autonomous driving: A review10.1109/TNNLS.2020.3015992 ↗
88. PI-RCNN: An efficient multi-sensor 3D object detector with point-based attentive cont-conv fusion module
89. Pointnet: Deep learning on point sets for 3D classification and segmentation
90. A multilevel multimodal fusion transformer for remote sensing semantic segmentation
91. Visual tracking in complex scenes: A location fusion mechanism based on the combination of multiple visual cognition flows10.1016/j.inffus.2023.02.005 ↗
92. Sensor fusion for joint 3D object detection and semantic segmentation
93. KDA3D: Key-point densification and multi-attention guidance for 3D object detection10.3390/rs12111895 ↗
94. SimDistill: Simulated multi-modal distillation for BEV 3D object detection
95. EPNet: Enhancing point features with image semantics for 3D object detection
96. Seeing through fog without seeing fog: Deep multi-modal sensor fusion in unseen adverse weather
97. RadarNet: Exploiting radar for robust perception of dynamic objects
98. Robust multi-modal vehicle detection in foggy weather using complementary lidar and radar signals
99. CLOCs: Camera-LiDAR object candidates fusion for 3D object detection
100. Fast and accurate 3D object detection for lidar-camera-based autonomous vehicles using one shared voxel-based backbone10.1109/ACCESS.2021.3055491 ↗