Road Extraction from Remote Sensing Images Using a Skip-Connected Parallel CNN-Transformer Encoder-Decoder Model
Source abstract
Abstract
Extracting roads from remote sensing images holds significant practical value across fields like urban planning, traffic management, and disaster monitoring. Current Convolutional Neural Network (CNN) methods, praised for their robust local feature learning enabled by inductive biases, deliver impressive results. However, they face challenges in capturing global context and accurately extracting the linear features of roads due to their localized receptive fields. To address these shortcomings of traditional methods, this paper proposes a novel parallel encoder architecture that integrates a CNN Encoder Module (CEM) with a Transformer Encoder Module (TEM). The integration combines the CEM’s strength in local feature extraction with the TEM’s ability to incorporate global context, achieving complementary advantages and overcoming limitations of both Transformers and CNNs. Furthermore, the architecture also includes a Linear Convolution Module (LCM), which uses linear convolutions tailored to the shape and distribution of roads. By capturing image features in four specific directions, the LCM significantly improves the model’s ability to detect and represent global and linear road features. Experimental results demonstrate that our proposed method achieves substantial improvements on the German-Street Dataset and the Massachusetts Roads Dataset, increasing the Intersection over Union (IoU) of road class by at least 3% and the overall F1 score by at least 2%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Automated Road and Building Extraction · Remote Sensing and LiDAR Applications · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Artificial Intelligence, Nanjing Agricultural University, Nanjing 210095, China
Innovation Academy for Microsatellites of CAS, Shanghai 201210, China
Shanghai Engineering Center for Microsatellites, Shanghai 201210, China
Key Laboratory for Satellite Digitalization Technology of CAS, Shanghai 201210, China
Funding
the National Defense Science and Technology Outstanding Youth Science Fund
References (41)
Show source-provided references
1. Path Planning for GEO-UAV Bistatic SAR Using Constrained Adaptive Multiobjective Differential Evolution10.1109/TGRS.2016.2585184 ↗
2. The Role of Artificial Intelligence in the Integration of Remotely Sensed Data with Geographic Information Systems10.1109/TGRS.1987.289804 ↗
3. Xu, W., Wei, J., Dolan, J.M., Zhao, H., and Zha, H. (2012, January 14–18). A Real-Time Motion Planner with Trajectory Optimization for Autonomous Vehicles. Proceedings of the 2012 IEEE International Conference on Robotics and Automation, Saint Paul, MN, USA.
4. A review of road extraction from remote sensing images
5. Road Extraction Methods in High-Resolution Remote Sensing Images: A Comprehensive Review10.1109/JSTARS.2020.3023549 ↗
6. Mo, S., Shi, Y., Yuan, Q., and Li, M. (2024). A Survey of Deep Learning Road Extraction Algorithms Using High-Resolution Remote Sensing Images. Sensors, 24.10.3390/s24051708 ↗
7. Anil, P.N., and Natarajan, S. (2010, January 9–11). A Novel Approach Using Active Contour Model for Semi-Automatic Road Extraction from High Resolution Satellite Imagery. Proceedings of the Second International Conference on Machine Learning & Computing, Bangalore, India.10.1109/ICMLC.2010.36 ↗
8. Abraham, L., and Sasikumar, M. (2013, January 22–25). A fuzzy based road network extraction from degraded satellite images. Proceedings of the 2013 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Mysore, India.10.1109/ICACCI.2013.6637494 ↗
9. Mnih, V., and Hinton, G.E. (2010). Learning to Detect Roads in High-Resolution Aerial Images, Springer.10.1007/978-3-642-15567-3_16 ↗
10. Fully Convolutional Networks for Semantic Segmentation
11. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs10.1109/TPAMI.2017.2699184 ↗
12. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
13. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv.10.1007/978-3-319-24574-4_28 ↗
14. Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N., and Liang, J. (2018). UNet++: A Nested U-Net Architecture for Medical Image Segmentation. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. DLMIA ML-CDS 2018, Springer.10.1007/978-3-030-00889-5_1 ↗
15. Jha, D., Smedsrud, P.H., Johansen, D., De Lange, T., Johansen, H.D., Halvorsen, P., and Riegler, M.A. (2021). A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation. arXiv.10.1109/JBHI.2021.3049304 ↗
16. Kumar, P., Nagar, P., Arora, C., and Gupta, A. (2018, January 7–10). U-SegNet: Fully Convolutional Neural Network based Automated Brain tissue segmentation Tool. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.10.1109/ICIP.2018.8451295 ↗
17. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation10.1109/TPAMI.2016.2644615 ↗
18. NL-LinkNet: Toward Lighter But More Accurate Road Extraction With Nonlocal Operations
19. Chaurasia, A., and Culurciello, E. (2017, January 10–13). LinkNet: Exploiting encoder representations for efficient semantic segmentation. Proceedings of the 2017 IEEE Visual Communications and Image Processing (VCIP), St. Petersburg, FL, USA.10.1109/VCIP.2017.8305148 ↗
20. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations, Virtual.
21. Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J., Tian, J., Zhang, Y., Shi, Z., Fan, J., and He, Z. (2021). A Survey of Visual Transformers. arXiv.
22. Han, K., Wang, Y., Chen, H., Chen, X., and Tao, D. (2023). A Survey on Visual Transformer. arXiv.
23. Efficient Transformers: A Survey10.1145/3530811 ↗
24. Zheng, S., Lu, J., Zhao, H., Zhu, X., and Zhang, L. (2020). Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers. arXiv.10.1109/CVPR46437.2021.00681 ↗
25. Wang, W., Xie, E., Li, X., Fan, D.P., and Shao, L. (2021). Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions. arXiv.10.1109/ICCV48922.2021.00061 ↗
26. Chu, X., Tian, Z., Wang, Y., Zhang, B., Ren, H., Wei, X., Xia, H., and Shen, C. (2021). Twins: Revisiting the Design of Spatial Attention in Vision Transformers. arXiv.
27. Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., and Liu, W. (November, January 27). CCNet: Criss-Cross Attention for Semantic Segmentation. Proceedings of the International Conference on Computer Vision, Seoul, Republic of Korea.
28. Xu, Z., Liu, Y., Gan, L., Sun, Y., Liu, M., and Wang, L. (2022). RNGDet: Road Network Graph Detection by Transformer in Aerial Images. arXiv.10.1109/TGRS.2022.3186993 ↗
29. Towards accurate and efficient road extraction by leveraging the characteristics of road shapes
30. Hetang, C., Xue, H., Le, C., Yue, T., Wang, W., and He, Y. (2024, January 17–18). Segment Anything Model for Road Network Graph Extraction. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.10.1109/CVPRW63382.2024.00262 ↗
31. Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv.
32. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
33. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
34. Islam, M.A., Jia, S., and Bruce, N.D.B. (2020). How Much Position Information Do Convolutional Neural Networks Encode?. arXiv.
35. Learning Aerial Image Segmentation From Online Maps10.1109/TGRS.2017.2719738 ↗
36. Mnih, V. (2013). Machine Learning for Aerial Image Labeling. [Ph.D. Thesis, University of Toronto].
37. Loshchilov, I., and Hutter, F. (2017). Decoupled Weight Decay Regularization. arXiv.
38. Chen, L.C., Papandreou, G., Schroff, F., and Adam, H. (2017). Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv.
39. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2016, January 21–26). Pyramid Scene Parsing Network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
40. Zhao, H., Zhang, Y., Liu, S., Shi, J., Loy, C.C., Lin, D., and Jia, J. (2018). PSANet: Point-Wise Spatial Attention Network for Scene Parsing, Springer.10.1007/978-3-030-01240-3_17 ↗
41. Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., and Luo, P. (2021). SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. arXiv.