Infrared–Visible Image Fusion via Cross-Modal Guided Dual-Branch Networks
Source abstract
Abstract
In the field of low-altitude aerial drone data fusion, the fusion of infrared and visible light images remains challenging due to issues such as large modal differences, insufficient cross-modal alignment, and limited global context modeling. Traditional methods struggle to extract complementary information across modalities, while deep learning methods often lack sufficient global receptive fields (convolutional neural networks) or fail to preserve local details (standard Transformers). To address these issues, we propose a Cross-modal Guided Dual-Branch Network (CGDBN) that combines convolutional neural networks and Transformer architecture. Our framework contribution: We designed a Target-modal Feature Extraction Mechanism (TMFEM) module with specialized thermal characteristics for infrared feature extraction, which does not require processing of visible light features; we introduced Simplified Linear Attention Blocks (SLABs) into our framework to improve global context capture as a module; we designed a Cross-Modal Interaction Mechanism (CMIM) module for bidirectional feature interaction; and we designed a Density Adaptive Multimodal Fusion (DAMF) module that weights modal contributions based on content analysis. This asymmetric design recognizes that different types of images have different characteristics and require targeted processing. The experimental results on AVMS, M3FD, and TNO datasets show that the proposed model has a peak signal-to-noise ratio (PSNR) of 16.2497 on the AVMS dataset, which is 0.9971 higher than the best benchmark method YDTR (peak signal-to-noise ratio: approximately 15.2526). The peak signal-to-noise ratio on the M3FD dataset is 16.5044, which is 0.7480 higher than the best benchmark method YDTR (peak signal-to-noise ratio of approximately 15.7564). The peak signal-to-noise ratio on the TNO dataset is 17.3956, which is 0.7934 higher than the best benchmark method YDTR (peak signal-to-noise ratio: approximately 16.6022), and the overall performance on all other indicators is among the top in all comparison models. This method has broad application prospects in fields such as drone data fusion.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Image Fusion Techniques · Image Enhancement Techniques · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
The 54th Research Institute of CETC, Shijiazhuang 050081, China
CETC Key Laboratory of Aerospace Information Applications, Shijiazhuang 050081, China
School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
References (43)
Show source-provided references
1. Multi-focus image fusion with a deep convolutional neural network10.1016/j.inffus.2016.12.001 ↗
2. Infrared and visible image fusion methods and applications: A survey10.1016/j.inffus.2018.02.004 ↗
3. Multimodal sensor medical image fusion based on type-2 fuzzy logic in NSCT domain10.1109/JSEN.2016.2533864 ↗
4. Gradient-based multiresolution image fusion10.1109/TIP.2004.823821 ↗
5. DCDR-GAN: A Densely Connected Disentangled Representation Generative Adversarial Network for Infrared and Visible Image Fusion10.1109/TCSVT.2022.3206807 ↗
6. Vibashan, V.S., Valanarasu, J.M.J., Oza, P., and Patel, V.M. (2022, January 16–19). Image Fusion Transformer. Proceedings of the International Conference on Image Processing (ICIP), Bordeaux, France.
7. Singh, R., and Khare, A. (2013, January 11–13). Multimodal medical image fusion using daubechies complex wavelet transform. Proceedings of the IEEE Conference on Information & Communication Technologies, Thuckalay, India.10.1109/CICT.2013.6558217 ↗
8. Li, P., Wang, H., Li, X., Hu, H., Wei, H., Yuan, Y., Zhang, Z., and Qi, G. (2019, January 3–5). A novel Image Fusion Framework based on Non-Subsampled Shearlet Transform (NSST) Domain. Proceedings of the Chinese Control and Decision Conference (CCDC), Nanchang, China.10.1109/CCDC.2019.8833211 ↗
9. Chen, Z., Zhang, C., and Wang, P. (2014, January 14–16). High-quality fusion for visible and infrared images based on the double NSCT. Proceedings of the 7th International Congress on Image and Signal Processing, Dalian, China.10.1109/CISP.2014.7003781 ↗
10. An image fusion algorithm based on image clustering theory10.1007/s00371-024-03736-1 ↗
11. Ting, L., and Jian, C. (2012, January 24–26). A new algorithm for image fusion via sparse representation. Proceedings of the International Conference on Automatic Control and Artificial Intelligence (ACAI 2012), Xiamen, China.
12. Tabarsaii, S., Aghagolzade, A., and Ezoji, M. (2019, January 6–9). Sparse Representation-based Multi-focus Image Fusion in a Hybrid of DWT and NSCT. Proceedings of the 4th International Conference on Pattern Recognition and Image Analysis (IPRIA), Tehran, Iran.10.1109/PRIA.2019.8785978 ↗
13. Hyperspectral and Multispectral Image Fusion via Nonlocal Low-Rank Tensor Approximation and Sparse Representation10.1109/TGRS.2020.2994968 ↗
14. A multi-source image fusion algorithm based on gradient regularized convolution sparse representation10.23919/JSEE.2020.000027 ↗
15. Prabhakar, K., Srikar, V.S., and Babu, R.V. (2017, January 22–29). DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.505 ↗
16. TCCL-DenseFuse: Infrared and Water Vapor Satellite Image Fusion Model Using Deep Learning10.1109/JSTARS.2023.3277842 ↗
17. Deep Selective Fusion of Visible and Near-Infrared Images Using Unsupervised U-Net10.1109/TNNLS.2022.3142780 ↗
18. Lv, H., Deng, B., and Li, X. (2022, January 12–14). Research on Image Fusion Technology of Infrared and Visible Image Based on MST and CNN. Proceedings of the IEEE 4th International Conference on Civil Aviation Safety and Information Technology (ICCASIT), Dali, China.10.1109/ICCASIT55263.2022.9986707 ↗
19. Shen, S., Li, D., Mei, L., Xu, C., Ye, Z., Zhang, Q., Hong, B., Yang, W., and Wang, Y. (2023). DFA-Net: Multi-Scale Dense Feature-Aware Network via Integrated Attention for Unmanned Aerial Vehicle Infrared and Visible Image Fusion. Drones, 7.10.3390/drones7080517 ↗
20. Le, Z., Huang, J., Fan, F., Tian, X., and Ma, J. (2020, January 25–28). A Generative Adversarial Network For Medical Image Fusion. Proceedings of the IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates.10.1109/ICIP40778.2020.9191089 ↗
21. Yanli, L., Zimu, L., Junce, F., and Gu, Y. (2022, January 11–12). An Unsupervised GAN-based Quality-enhanced Medical Image Fusion Network. Proceedings of the IEEE Conference on Telecommunications, Optics and Computer Science (TOCS 2022), Dalian, China.10.1109/TOCS56154.2022.10016141 ↗
22. VIF-Net: An Unsupervised Framework for Infrared and Visible Image Fusion
23. Efficient feature difference-based infrared and visible image fusion for low-light environments10.1007/s00371-025-03841-9 ↗
24. Dong, G., and Basu, A. (2024). Medical image denoising via explainable AI feature preserving loss. arXiv.10.1007/978-3-031-82475-3_2 ↗
25. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. arXiv.
26. DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation
27. SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer10.1109/JAS.2022.105686 ↗
28. Jia, D., Guo, J., Han, K., Wu, H., Zhang, C., Xu, C., and Chen, X. (2024). GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer. arXiv.
29. Zamir, S.W., Arora, A., Khan, S.H., Hayat, M., Khan, F.S., and Yang, M. (2022, January 18–24). Restormer: Efficient Transformer for High-Resolution Image Restoration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00564 ↗
30. EMFusion: An unsupervised enhanced medical image fusion network10.1016/j.inffus.2021.06.001 ↗
31. CTFusion: CNN-transformer-based self-supervised learning for infrared and visible image fusion10.3934/mbe.2024294 ↗
32. Wang, P., Zhou, Y., Yang, G., and Wu, H. (2024, January 20–22). ResSCFuse: A Fusion Network for Infrared and Visible Images Based on Restormer Attention and Semi-Coupled Feature Reconstruction Net. Proceedings of the 4th International Conference on Digital Society and Intelligent Systems, Sydney, Australia.10.1109/DSInS64146.2024.10992108 ↗
33. Cross-Modal Transformers for Infrared and Visible Image Fusion10.1109/TCSVT.2023.3289170 ↗
34. Zhao, Z., Bai, H., Zhang, J., Zhang, Y., Xu, S., Lin, Z., Timofte, R., and Van Gool, L. (2023, January 17–24). CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00572 ↗
35. Wang, X., Zhu, C., and Li, J. (July, January 30). CTUnet: A Novel Paradigm Integrating CNNs and Transformers for Medical Image Segmentation. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2024), Yokohama, Japan.10.1109/IJCNN60899.2024.10650785 ↗
36. CPIFuse: Toward realistic color and enhanced textures in color polarization image fusion10.1016/j.inffus.2025.103111 ↗
37. Guo, J., Chen, X., Tang, Y., and Wang, Y. (2024). SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization. arXiv.
38. YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer10.1109/TMM.2022.3192661 ↗
39. FusionGAN: A generative adversarial network for infrared and visible image fusion10.1016/j.inffus.2018.09.004 ↗
40. ITFuse: An interactive transformer for infrared and visible image fusion10.1016/j.patcog.2024.110822 ↗
41. FS-Diff: Semantic guidance and clarity-aware simultaneous multimodal image fusion and super-resolution10.1016/j.inffus.2025.103146 ↗
42. Liu, J., Fan, X., Huang, Z., Wu, G., Liu, R., Zhong, W., and Luo, Z. (2022, January 18–24). Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00571 ↗
43. Alexander, T. (2014). TNO Image Fusion Dataset, TNO.