Research on the Applicability of Transformer Model in Remote-Sensing Image Segmentation
Source abstract
Abstract
Transformer models have achieved great results in the field of computer vision over the past 2 years, drawing attention from within the field of remote sensing. However, there are still relatively few studies on this model in the field of remote sensing. Which method is more suitable for remote-sensing segmentation? In particular, how do different transformer models perform in the face of high-spatial resolution and the multispectral resolution of remote-sensing images? To explore these questions, this paper presents a comprehensive comparative analysis of three mainstream transformer models, including the segmentation transformer (SETRnet), SwinUnet, and TransUnet, by evaluating three aspects: a visual analysis of feature-segmentation results, accuracy, and training time. The experimental results show that the transformer structure has obvious advantages for the feature-extraction ability of large-scale remote-sensing data sets and ground objects, but the segmentation performance of different transfer structures in different scales of remote-sensing data sets is also very different. SwinUnet exhibits better global semantic interaction and pixel-level segmentation prediction on the large-scale Potsdam data set, and the SwinUnet model has the highest accuracy metrics for KAPPA, MIoU, and OA in the Potsdam data set, at 76.47%, 63.62%, and 85.01%, respectively. TransUnet has better segmentation results in the small-scale Vaihingen data set, and the three accuracy metrics of KAPPA, MIoU, and OA are the highest, at 80.54%, 56.25%, and 85.55%, respectively. TransUnet is better able to handle the edges and details of feature segmentation thanks to the network structure together built by its transformer and convolutional neural networks (CNNs). Therefore, TransUnet segmentation accuracy is higher when using a small-scale Vaihingen data set. Compared with SwinUnet and TransUnet, the segmentation performance of SETRnet in different scales of remote-sensing data sets is not ideal, so SETRnet is not suitable for the research task of remote-sensing image segmentation. In addition, this paper discusses the reasons for the performance differences between transformer models and discusses the differences between transformer models and CNN. This study further promotes the application of transformer models in remote-sensing image segmentation, improves the understanding of transformer models, and helps relevant researchers to select a more appropriate transformer model or model improvement method for remote-sensing image segmentation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Remote-Sensing Image Classification · Advanced Neural Network Applications · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
The College of Geography and Environment Science, Henan University, Kaifeng 475004, China
Key Laboratory of Geospatial Technology for Middle and Lower Yellow River Regions, Ministry of Education, Kaifeng 475004, China
Henan Technology Innovation Center of Spatio-Temporal Big Data, Henan University, Zhengzhou 450046, China
Henan Industrial Technology Academy of Spatio-Temporal Big Data, Henan University, Zhengzhou 450046, China
Funding
State Administration of Science, Technology, and Industry for the National Defense of PRC · National Science and Technology Platform Construction · Key Projects of the National Regional Innovation Joint Fund · Ministry of Education · Open Program of Collaborative Innovation Center of Geo-Information Technology for Smart Central Plains Henan Province
References (43)
Show source-provided references
1. MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation10.1109/JBHI.2021.3138024 ↗
2. Olive Fly Sting Detection Based on Computer Vision10.1016/j.postharvbio.2019.01.003 ↗
3. Meta-Learning Based Prediction of Different Corn Cultivars from Colour Feature Extraction with Image Processing Technique10.15832/ankutbd.567407 ↗
4. Remote Sensing Image Segmentation by Combining Spectral and Texture Features10.1109/TGRS.2012.2234755 ↗
5. Remote Sensing Image Segmentation Advances: A Meta-Analysis10.1016/j.isprsjprs.2021.01.020 ↗
6. Image Segmentation Methods Based on Superpixel Techniques: A Survey
7. Xiong, D., He, C., Liu, X., and Liao, M. (2020). An End-To-End Bayesian Segmentation Network Based on a Generative Adversarial Network for Remote Sensing Images. Remote Sens., 12.10.3390/rs12020216 ↗
8. Zheng, K., Wang, H., Qin, F., and Han, Z. (2022). A Land Use Classification Model Based on Conditional Random Fields and Attention Mechanism Convolutional Networks. Remote Sens., 14.10.3390/rs14112688 ↗
9. Misbah, K., Laamrani, A., Khechba, K., Dhiba, D., and Chehbouni, A. (2021). Multi-Sensors Remote Sensing Applications for Assessing, Monitoring, and Mapping NPK Content in Soil and Crops in African Agricultural Land. Remote Sens., 14.10.3390/rs14010081 ↗
10. Sataer, G., Sultan, M., Emil, M.K., Yellich, J.A., Palaseanu-Lovejoy, M., Becker, R., Gebremichael, E., and Abdelmohsen, K. (2022). Remote Sensing Application for Landslide Detection, Monitoring along Eastern Lake Michigan (Miami Park, MI). Remote Sens., 14.10.3390/rs14143474 ↗
11. Joint Deep Learning for Land Cover and Land Use Classification10.1016/j.rse.2018.11.014 ↗
12. Challenges in Using Land Use and Land Cover Data for Global Change Studies10.1111/j.1365-2486.2010.02307.x ↗
13. Geographic Object-Based Image Analysis—Towards a New Paradigm10.1016/j.isprsjprs.2013.09.014 ↗
14. Scale Parameter Selection by Spatial Statistics for GeOBIA: Using Mean-Shift Based Multi-Scale Segmentation as an Example10.1016/j.isprsjprs.2015.04.010 ↗
15. Talukdar, S., Singha, P., Mahato, S., Pal, S., Liou, Y.-A., and Rahman, A. (2020). Land-Use Land-Cover Classification by Machine Learning Classifiers for Satellite Observations—A Review. Remote Sens., 12.10.3390/rs12071135 ↗
16. Support Vector Machine Versus Random Forest for Remote Sensing Image Classification: A Meta-Analysis and Systematic Review10.1109/JSTARS.2020.3026724 ↗
17. Remote Sensing Image Classification: A Survey of Support-Vector-Machine-Based Advanced Techniques10.1109/MGRS.2016.2641240 ↗
18. Random Forest and Rotation Forest for Fully Polarized SAR Image Classification Using Polarimetric and Spatial Features10.1016/j.isprsjprs.2015.03.002 ↗
19. Vehicle Detection from UAV Imagery with Deep Learning: A Review
20. Machine Learning Techniques for Biomedical Image Segmentation: An Overview of Technical Aspects and Introduction to State-of-art Applications10.1002/mp.13649 ↗
21. Alem, A., and Kumar, S. (2020, January 4–5). Deep Learning Methods for Land Cover and Land Use Classification in Remote Sensing: A Review. Proceedings of the 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India.10.1109/ICRITO48877.2020.9197824 ↗
22. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc., Lake Tahoe, NV, USA.
23. A2-FPN for Semantic Segmentation of Fine-Resolution Remotely Sensed Images10.1080/01431161.2022.2030071 ↗
24. Automatic Building Extraction from High-Resolution Aerial Images and LiDAR Data Using Gated Residual Refinement Network10.1016/j.isprsjprs.2019.02.019 ↗
25. Urban Land-Use Mapping Using a Deep Convolutional Neural Network with High Spatial Resolution Multispectral Remote Sensing Imagery10.1016/j.rse.2018.04.050 ↗
26. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. arXiv.10.1007/978-3-030-01234-2_49 ↗
27. Navab, N., Hornegger, J., Wells, W.M., and Frangi, A.F. (2015). Proceedings of the Medical Image Computing and Computer-Assisted Intervention—MICCAI, Munich, Germany, 5–9 October 2015, Springer International Publishing.10.1007/978-3-319-24553-9 ↗
28. Li, H., Xiong, P., An, J., and Wang, L. (2018). Pyramid Attention Network for Semantic Segmentation. arXiv.
29. Wang, W., Zhou, T., Yu, F., Dai, J., Konukoglu, E., and Van Gool, L. (2021, January 10–17). Exploring Cross-Image Pixel Contrast for Semantic Segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00721 ↗
30. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2022). An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.
31. Aleissaee, A.A., Kumar, A., Anwer, R.M., Khan, S., Cholakkal, H., Xia, G.-S., and Khan, F.S. (2022). Transformers in Remote Sensing: A Survey. arXiv.
32. Non-Autoregressive Transformer for Speech Recognition10.1109/LSP.2020.3044547 ↗
33. Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., and Torr, P.H.S. (2021, January 20–25). Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00681 ↗
34. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10–17). Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00986 ↗
35. Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., and Zhou, Y. (2021). TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation. arXiv.
36. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
37. Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., and Wang, M. (2021). Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation. arXiv.
38. Image Segmentation Evaluation: A Survey of Unsupervised Methods10.1016/j.cviu.2007.08.003 ↗
39. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs10.1109/TPAMI.2017.2699184 ↗
40. Chattopadhyay, S., and Basak, H. (2020). Multi-Scale Attention U-Net (MsAUNet): A Modified U-Net Architecture for Scene Segmentation. arXiv.
41. Sun, Z., Zhou, W., Ding, C., and Xia, M. (2022). Multi-Resolution Transformer Network for Building and Road Segmentation of Remote Sensing Image. ISPRS Int. J. Geo-Inf., 11.10.3390/ijgi11030165 ↗
42. Yao, J., and Jin, S. (2022). Multi-Category Segmentation of Sentinel-2 Images Based on the Swin UNet Method. Remote Sens., 14.10.3390/rs14143382 ↗
43. Swin Transformer Embedding UNet for Remote Sensing Image Semantic Segmentation10.1109/TGRS.2022.3230846 ↗