Multimodal Image Segmentation with Dynamic Adaptive Window and Cross-Scale Fusion for Heterogeneous Data Environments
Source abstract
Abstract
Multi-modal image segmentation is a key task in various fields such as urban planning, infrastructure monitoring, and environmental analysis. However, it remains challenging due to complex scenes, varying object scales, and the integration of heterogeneous data sources (such as RGB, depth maps, and infrared). To address these challenges, we proposed a novel multi-modal segmentation framework, DyFuseNet, which features dynamic adaptive windows and cross-scale feature fusion capabilities. This framework consists of three key components: (1) Dynamic Window Module (DWM), which uses dynamic partitioning and continuous position bias to adaptively adjust window sizes, thereby improving the representation of irregular and fine-grained objects; (2) Scale Context Attention (SCA), a hierarchical mechanism that associates local details with global semantics in a coarse-to-fine manner, enhancing segmentation accuracy in low-texture or occluded regions; and (3) Hierarchical Adaptive Fusion Architecture (HAFA), which aligns and fuses features from multiple modalities through shallow synchronization and deep channel attention, effectively balancing complementarity and redundancy. Evaluated on benchmark datasets (such as ISPRS Vaihingen and Potsdam), DyFuseNet achieved state-of-the-art performance, with mean Intersection over Union (mIoU) scores of 80.40% and 80.85%, surpassing MFTransNet by 1.91% and 1.77%, respectively. The model also demonstrated strong robustness in challenging scenes (such as building edges and shadowed objects), achieving an average F1 score of 85% while maintaining high efficiency (26.19 GFLOPs, 30.09 FPS), making it suitable for real-time deployment. This work presents a practical, versatile, and computationally efficient solution for multi-modal image analysis, with potential applications beyond remote sensing, including smart monitoring, industrial inspection, and multi-source data fusion tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Medical Image Segmentation Techniques · Image Retrieval and Classification Techniques · Advanced Image Fusion Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information and Control Engineering, Xi’an University of Architecture and Technology, Xi’an 710055, China
Institute for Interdisciplinary and Innovate Research, Xi’an University of Architecture and Technology, Xi’an 710055, China
Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China
Funding
National Key Research and Development Plan · Humanities and Social Science Research Project of the Ministry of Education of China · Cultivation Special Project for Frontier Interdisciplinary Fields of Xi’an University of Architecture and Technology
References (30)
Show source-provided references
1. Li, K., Liu, R., Cao, X., Bai, X., Zhou, F., Meng, D., and Wang, Z. (2025, January 11–15). SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR52734.2025.00986 ↗
2. State Council (2021). National Medium- and Long-Term Plan for Basic Surveying and Mapping (2021–2030), Standards Press of China.
3. Cross-Domain Few-Shot Contrastive Learning for Hyperspectral Images Classification
4. Real-Time Urban Street View Semantic Segmentation Based on Cross-Level Aggregation Network10.37188/OPE.20243208.1212 ↗
5. Geological Mapping via Convolutional Neural Network Based on Remote Sensing and Geochemical Survey Data in Vegetation Coverage Areas10.1109/JSTARS.2023.3260584 ↗
6. AED-Net: A Semantic Segmentation Model for Landslide Disaster Remote Sensing Images
7. A Traffic Scene Perception Algorithm Combining Semantic Segmentation and Depth Estimation
8. Evaluation and Influencing Factors Analysis of Road Environment Safety Perception Combining Street View Images and Machine Learning
9. Improved U-Net++ Semantic Segmentation Method for Remote Sensing Images10.1109/ACCESS.2025.3552581 ↗
10. Combining Swin Transformer With UNet for Remote Sensing Image Semantic Segmentation
11. Fusion PSPnet Image Segmentation Based Method for Multi-Focus Image Fusion
12. Deep Hierarchical Vision Transformer for Hyperspectral and LiDAR Data Classification10.1109/TIP.2022.3162964 ↗
13. Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., and Torr, P.H.S. (2021, January 20). Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00681 ↗
14. Karlinsky, L., Michaeli, T., and Nishino, K. (2022, January 23–27). Swin-Unet: Unet-Like Pure Transformer for Medical Image Segmentation. Proceedings of the Computer Vision–ECCV 2022 Workshops, Tel Aviv, Israel.10.1007/978-3-031-25056-9 ↗
15. Development Status and Future Prospects of Multi-Source Remote Sensing Image Fusion10.11834/jrs.20210259 ↗
16. Densely Based Multi-Scale and Multi-Modal Fully Convolutional Networks for High-Resolution Remote-Sensing Image Semantic Segmentation10.1109/JSTARS.2019.2906387 ↗
17. A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data10.1109/JSTARS.2022.3165005 ↗
18. Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data10.1109/TGRS.2022.3231215 ↗
19. A Multimodal Hyper-Fusion Transformer for Remote Sensing Image Classification10.1016/j.inffus.2023.03.005 ↗
20. ObjFormer: Learning Land-Cover Changes From Paired OSM Data and Optical High-Resolution Imagery via Object-Guided Transformer
21. A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation
22. Deep High-Resolution Representation Learning for Visual Recognition10.1109/TPAMI.2020.2983686 ↗
23. Hu, X., Yang, K., Fei, L., and Wang, K. (2019, January 22). ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8803025 ↗
24. He, S., Yang, H., Zhang, X., and Li, X. (2023). MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images. Mathematics, 11.10.3390/math11030722 ↗
25. STransFuse: Fusing Swin Transformer and Convolutional Neural Network for Remote Sensing Image Semantic Segmentation10.1109/JSTARS.2021.3119654 ↗
26. Adjacent-Scale Multimodal Fusion Networks for Semantic Segmentation of Remote Sensing Data10.1109/JSTARS.2024.3486906 ↗
27. Ren, P., Li, C., Wang, G., Xiao, Y., Du, Q., Liang, X., and Chang, X. (2022, January 18). Beyond Fixation: Dynamic Window Visual Transformer. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01168 ↗
28. Zhang, Z., Shu, D., Gu, G., Hu, W., Wang, R., Chen, X., and Yang, B. (2025). RingFormer-Seg: A Scalable and Context-Preserving Vision Transformer Framework for Semantic Segmentation of Ultra-High-Resolution Remote Sensing Imagery. Remote Sens., 17.10.3390/rs17173064 ↗
29. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., and Hu, Q. (2020, January 13). ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01155 ↗
30. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2016). Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. arXiv.10.1609/aaai.v31i1.11231 ↗