A Novel Global-Local Feature Aggregation Framework for Semantic Segmentation of Large-Format High-Resolution Remote Sensing Images
Source abstract
Abstract
In high-resolution remote sensing images, there are areas with weak textures such as large building roofs, which occupy a large number of pixels in the image. These areas pose a challenge for traditional semantic segmentation networks to obtain ideal results. Common strategies like downsampling, patch cropping, and cascade models often sacrifice fine details or global context, resulting in limited accuracy. To address these issues, a novel semantic segmentation framework has been designed specifically for large-format high-resolution remote sensing images by aggregating global and local features in this paper. The framework consists of two branches: one branch deals with low-resolution downsampled images to capture global features, while the other branch focuses on cropped patches to extract high-resolution local details. Also, this paper introduces a feature aggregation module based on the Transformer structure, which effectively aggregates global and local information. Additionally, to save GPU memory usage, a novel three-step training method has been developed. Extensive experiments on two public datasets demonstrate the effectiveness of the proposed approach, with an IoU of 90.83% on the AIDS dataset and 90.30% on the WBDS dataset, surpassing state-of-the-art methods such as DANet, DeepLab v3+, U-Net, ViT, TransUNet, CMTFNet, and UANet.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Remote-Sensing Image Classification · Advanced Image and Video Retrieval Techniques · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
China Aero Geophysical Survey and Remote Sensing Center for Natural Resources, Beijing 100083, China
College of Informatics, Huazhong Agricultural University, Wuhan 430070, China
Faculty of Artificial Intelligence in Education, Central China Normal University, 152 Luoyu Road, Wuhan 430079, China
Funding
National Key Research and Development Program of China · Ministry of Education of the People’s Republic of China · Knowledge Innovation Program of Wuhan-Shuguang Project · University-Industry Collaborative Education Program · Fundamental Research Funds for the Central Universities
References (35)
Show source-provided references
1. Advances in Hyperspectral Image Classification: Earth Monitoring with Statistical Learning Methods10.1109/MSP.2013.2279179 ↗
2. Fully Convolutional Networks for Semantic Segmentation10.1109/TPAMI.2016.2572683 ↗
3. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, Cham, Switzerland.10.1007/978-3-319-24574-4_28 ↗
4. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs10.1109/TPAMI.2017.2699184 ↗
5. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Proceedings of the Computer Vision—ECCV 2018, Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
6. Chen, L.-C., Papandreou, G., Schroff, F., and Adam, H. (2017). Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv.
7. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid Scene Parsing Network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
8. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
9. Huang, G., Liu, Z., Maaten, L.V.D., and Weinberger, K.Q. (2017, January 21–26). Densely Connected Convolutional Networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
10. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2019, January 15–20). Dual Attention Network for Scene Segmentation. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00326 ↗
11. LANet: Local Attention Embedding to Improve the Semantic Segmentation of Remote Sensing Images10.1109/TGRS.2020.2994150 ↗
12. Song, W., Zhou, X., Zhang, S., Wu, Y., and Zhang, P. (2023). GLF-Net: A Semantic Segmentation Model Fusing Global and Local Features for High-Resolution Remote Sensing Images. Remote Sens., 15.10.3390/rs15194649 ↗
13. Li, Y., Liu, Z., Yang, J., and Zhang, H. (2023). Wavelet Transform Feature Enhancement for Semantic Segmentation of Remote Sensing Images. Remote Sens., 15.10.3390/rs15245644 ↗
14. Multilevel Deformable Attention-Aggregated Networks for Change Detection in Bitemporal Remote Sensing Imagery
15. Li, X., Xu, F., Yong, X., Chen, D., Xia, R., Ye, B., Gao, H., Chen, Z., and Lyu, X. (2023). SSCNet: A Spectrum-Space Collaborative Network for Semantic Segmentation of Remote Sensing Images. Remote Sens., 15.10.3390/rs15235610 ↗
16. MACU-Net for Semantic Segmentation of Fine-Resolution Remotely Sensed Images
17. Depth-Assisted ResiDualGAN for Cross-Domain Aerial Images Semantic Segmentation
18. Xi, Z., Meng, Y., Chen, J., Deng, Y., Liu, D., Kong, Y., and Yue, A. (2023). Learning to Adapt Adversarial Perturbation Consistency for Domain Adaptive Semantic Segmentation of Remote Sensing Images. Remote Sens., 15.10.3390/rs15235498 ↗
19. SPGAN-DA: Semantic-Preserved Generative Adversarial Network for Domain Adaptive Remote Sensing Image Semantic Segmentation
20. A Self-Learning-Update CNN Model for Semantic Segmentation of Remote Sensing Images
21. PEGNet: Progressive Edge Guidance Network for Semantic Segmentation of Remote Sensing Images10.1109/LGRS.2020.2983464 ↗
22. Semantic Category Balance-Aware Involved Anti-Interference Network for Remote Sensing Semantic Segmentation
23. High-Order Semantic Decoupling Network for Remote Sensing Image Semantic Segmentation10.1109/TGRS.2023.3336665 ↗
24. UANet: An Uncertainty-Aware Network for Building Extraction From Remote Sensing Images
25. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
26. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.
27. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Virtual.10.1109/ICCV48922.2021.00986 ↗
28. Zhang, W., Huang, Z., Luo, G., Chen, T., Wang, X., Liu, W., Yu, G., and Shen, C. (2022, January 18–24). TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01177 ↗
29. SiamixFormer: A fully-transformer Siamese network with temporal Fusion for accurate building detection and change detection in bi-temporal remote sensing images10.1080/01431161.2023.2225228 ↗
30. Chen, J., Lu, Y., Yu, Q., Luo, X., and Zhou, Y. (2021). TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation. arXiv.
31. CMTFNet: CNN and Multiscale Transformer Fusion Network for Remote-Sensing Image Semantic Segmentation
32. STransFuse: Fusing Swin Transformer and Convolutional Neural Network for Remote Sensing Image Semantic Segmentation10.1109/JSTARS.2021.3119654 ↗
33. Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., and Xu, C. (2022, January 18–24). CMT: Convolutional Neural Networks Meet Vision Transformers. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01186 ↗
34. A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data10.1109/JSTARS.2022.3165005 ↗
35. Fully Convolutional Networks for Multisource Building Extraction From an Open Aerial and Satellite Imagery Data Set10.1109/TGRS.2018.2858817 ↗