GLFFNet: A Global and Local Features Fusion Network with Biencoder for Remote Sensing Image Segmentation
Source abstract
Abstract
In recent years, semantic segmentation of high-resolution remote sensing images has been gradually applied to many important scenes. However, with the rapid development of remote sensing data acquisition technology, the existing image data processing methods are facing major challenges. Especially in the accuracy of extraction and the integrity of the edges of objects, there are often problems such as small objects being assimilated by large objects. In order to solve the above problems, based on the excellent performance of Transformer, convolution and its variants, and feature pyramids in the field of deep learning image segmentation, we designed two encoders with excellent performance to extract global high-order interactive features and low-order local feature information. These encoders are then used as the backbone to construct a global and local feature fusion network with a dual encoder (GLFFNet) to effectively complete the segmentation of remote sensing images. Furthermore, a new auxiliary training module is proposed that uses the semantic attention layer to process the extracted feature maps separately, adjust the losses, and more specifically optimize each encoder of the backbone, thus optimizing the training process of the entire network. A large number of experiments show that our model achieves 87.96% mIoU on the Potsdam dataset and 80.42% mIoU on the GID dataset, and it has superior performance compared with some state-of-the-art methods on semantic segmentation tasks in the field of remote sensing.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Neural Network Applications · Remote-Sensing Image Classification · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Technology, North China University of Technology, Beijing 100144, China
Funding
National key research and development program of China
References (42)
Show source-provided references
1. A Review of Deep Learning Methods for Semantic Segmentation of Remote Sensing Imagery10.1016/j.eswa.2020.114417 ↗
2. ResUNet-a: A Deep Learning Framework for Semantic Segmentation of Remotely Sensed Data10.1016/j.isprsjprs.2020.01.013 ↗
3. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany. Proceedings, Part III 18.
4. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2015, January 7–9). Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
5. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs10.1109/TPAMI.2017.2699184 ↗
6. Chen, L.-C., Zhu, Y., Wang, H., Dabagia, M., Cheng, B., Li, Y., Liu, S., Adam, H., and Yuille, A.L. (2021). DeepLab2: A TensorFlow Library for Deep Labeling. arXiv.
7. Chen, L.-C., Papandreou, G., Schroff, F., and Adam, H. (2017, January 22–29). Rethinking Atrous Convolution for Semantic Image Segmentation. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.
8. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
9. HA U-Net: Improved Model for Building Extraction From High Resolution Remote Sensing Imagery10.1109/ACCESS.2021.3097630 ↗
10. Liu, R., Tao, F., Liu, X., Na, J., Leng, H., Wu, J., and Zhou, T. (2022). RAANet: A Residual ASPP with Attention Framework for Semantic Segmentation of High-Resolution Remote Sensing Images. Remote Sens., 14.10.3390/rs14133109 ↗
11. Chen, Z., Li, D., Fan, W., Guan, H., Wang, C., and Li, J. (2021). Self-Attention in Reconstruction Bias U-Net for Semantic Segmentation of Building Rooftops in Optical Remote Sensing Images. Remote Sens., 13.10.3390/rs13132524 ↗
12. Huang, L., Zhu, J., Qiu, M., Li, X., and Zhu, S. (2022). CA-BASNet: A Building Extraction Network in High Spatial Resolution Remote Sensing Images. Sustainability, 14.10.3390/su141811633 ↗
13. Zhang, Z., Xu, Z., Liu, C., Tian, Q., and Wang, Y. (2022). Cloudformer: Supplementary Aggregation Feature and Mask-Classification Network for Cloud Detection. Appl. Sci., 12.10.3390/app12073221 ↗
14. Zhang, Z., Xu, Z., Liu, C., Tian, Q., and Zhou, Y. (2022). Cloudformer V2: Set Prior Prediction and Binary Mask Weighted Network for Cloud Detection. Mathematics, 10.10.3390/math10152710 ↗
15. Zhang, Z., Miao, C., Liu, C., Tian, Q., and Zhou, Y. (2022). HA-RoadFormer: Hybrid Attention Transformer with Multi-Branch for Large-Scale High-Resolution Dense Road Segmentation. Mathematics, 10.10.3390/math10111915 ↗
16. Ziaee, A., Dehbozorgi, R., and Döller, M. (2021). A Novel Adaptive Deep Network for Building Footprint Segmentation. arXiv.
17. Chen, M., Wu, J., Liu, L., Zhao, W., Tian, F., Shen, Q., Zhao, B., and Du, R. (2021). DR-Net: An Improved Network for Building Extraction from High Resolution Remote Sensing Image. Remote Sens., 13.10.3390/rs13020294 ↗
18. An Attention-Fused Network for Semantic Segmentation of Very-High-Resolution Remote Sensing Imagery10.1016/j.isprsjprs.2021.05.004 ↗
19. UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery10.1016/j.isprsjprs.2022.06.008 ↗
20. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention Is All You Need. Proceedings of the Neural Information Processing Systems (NIPS), Long Beach, CA, USA.
21. Lu, Y., Wu, J., Shen, C., and van den Hengel, A. (2020, January 7–12). Gated Convolutional Networks with Hybrid Connectivity for Image Classification. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), New York, NY, USA.
22. Rao, Y., Lu, J., Zhou, J., and Tian, Q. (May, January 25). HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions. Proceedings of the International Conference on Learning Representations (ICLR), Virtual Event.
23. Lin, M., Chen, Q., and Yan, S. (2014, January 14–16). Network In Network. Proceedings of the International Conference on Learning Representations (ICLR), Banff, AB, Canada.
24. Fukui, H., Hirakawa, T., Yamashita, T., and Fujiyoshi, H. (2019, January 16–20). Attention Branch Network: Learning of Attention Mechanism for Visual Explanation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01096 ↗
25. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the International Conference on Machine Learning (ICML), Lille, France.
26. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Minneapolis, MN, USA.
27. Cheng, B., and Schwing, A.G. (2021, January 6–14). Per-Pixel Classification is Not All You Need for Semantic Segmentation. Proceedings of the Advances in Neural Information Processing Systems 34 (NeurIPS 2021), Virtual Event.
28. Song, Y., and Yan, H. (2017). Image Segmentation Algorithms Overview. arXiv.10.1109/AMS.2017.24 ↗
29. Thoma, M. (2016). A Survey of Semantic Segmentation. arXiv.
30. A Survey on Image Semantic Segmentation Using Deep Learning Techniques
31. Chen, X., Ding, M., Wang, X., Xin, Y., Mo, S., Wang, Y., and Wang, J. (2022). Context Autoencoder for Self-Supervised Representation Learning. arXiv.
32. ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network
33. Wang, L., Li, R., Wang, D., Duan, C., Wang, T., and Meng, X. (2021). Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images. Remote Sens., 13.10.3390/rs13163065 ↗
34. Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., and Luo, P. (2021). SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. arXiv.
35. Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv.
36. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., and Girdhar, R. (2021). Masked-attention Mask Transformer for Universal Image Segmentation. arXiv.10.1109/CVPR52688.2022.00135 ↗
37. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. arXiv.10.1109/ICCV48922.2021.00986 ↗
38. ImageNet Classification with Deep Convolutional Neural Networks10.1145/3065386 ↗
39. Crafting GBD-Net for Object Detection
40. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
41. Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
42. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations (ICLR), Virtual Event, Austria.