Dual-Branch Transformer–CNN Fusion for Enhanced Cloud Segmentation in Remote Sensing Imagery
Source abstract
Abstract
Cloud coverage and obstruction significantly affect the usability of remote sensing images, making cloud detection a key prerequisite for optical remote sensing applications. In existing cloud detection methods, using U-shaped convolutional networks alone has limitations in modeling long-range contexts, while Vision Transformers fall short in capturing local spatial features. To address these issues, this study proposes a dual-branch framework, TransCNet, which combines Transformer and CNN architectures to enhance the accuracy and effectiveness of cloud detection. TransCNet addresses this by designing dual encoder branches: a Transformer branch capturing global dependencies and a CNN branch extracting local details. A novel feature aggregation module enables the complementary fusion of multi-level features from both branches at each encoder stage, enhanced by channel attention mechanisms. To mitigate feature dilution during decoding, aggregated features compensate for information loss from sampling operations. Evaluations on 38-Cloud, SPARCS, and a high-resolution Landsat-8 dataset demonstrate TransCNet’s competitive performance across metrics, effectively balancing global semantic understanding and local edge preservation for clearer cloud boundary detection. The approach resolves key limitations in existing cloud detection frameworks through synergistic multi-branch feature integration.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Remote-Sensing Image Classification · Advanced Image Fusion Techniques · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Microelectronics Industry-Education Integration, Lanzhou University of Technology, Lanzhou 730050, China
School of Automation and Electrical Engineering, Lanzhou University of Technology, Lanzhou 730050, China
Funding
Scientific and Technological Project of Gansu Province · Scientific and Technological Project of Lanzhou City · Innovation Fund Project of Gansu Provincial Department of Education
References (52)
Show source-provided references
1. A multi-temporal method for cloud detection, applied to FORMOSAT-2, VENµS, LANDSAT and SENTINEL-2 images10.1016/j.rse.2010.03.002 ↗
2. Improvement and expansion of the Fmask algorithm: Cloud, cloud shadow, and snow detection for Landsats 4–7, 8, and Sentinel 2 images10.1016/j.rse.2014.12.014 ↗
3. Scientific basis and initial evaluation of the CLAVR-1 global clear/cloud classification algorithm for the Advanced Very High Resolution Radiometer10.1175/1520-0426(1999)016<0656:SBAIEO>2.0.CO;2 ↗
4. Characterization of the Landsat-7 ETM+ automated cloud-cover assessment (ACCA) algorithm10.14358/PERS.72.10.1179 ↗
5. Mohajerani, S., and Saeedi, P. (August, January 28). Cloud-Net: An end-to-end cloud detection algorithm for Landsat 8 imagery. Proceedings of the IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, Yokohama, Japan.10.1109/IGARSS.2019.8898776 ↗
6. Cloud and cloud shadow segmentation for remote sensing imagery via filtered jaccard loss function and parametric augmentation10.1109/JSTARS.2021.3070786 ↗
7. Hou, Q., Cheng, M.M., Hu, X., Borji, A., Tu, Z., and Torr, P.H. (2017, January 21–26). Deeply supervised salient object detection with short connections. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.563 ↗
8. CDnet: CNN-based cloud detection for remote sensing imagery10.1109/TGRS.2019.2904868 ↗
9. CDnetV2: CNN-based cloud detection for remote sensing imagery with cloud-snow coexistence10.1109/TGRS.2020.2991398 ↗
10. Dual-branch Network for Cloud and Cloud Shadow Segmentation
11. Zhang, Z., Xu, Z., Liu, C.A., Tian, Q., and Wang, Y. (2022). Cloudformer: Supplementary aggregation feature and mask-classification network for cloud detection. Appl. Sci., 12.10.3390/app12073221 ↗
12. Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., and Xia, H. (2021, January 19–25). End-to-end video instance segmentation with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00863 ↗
13. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
14. Hu, K., Zhang, D., and Xia, M. (2021). CDUNet: Cloud Detection UNet for Remote Sensing Imagery. Remote Sens., 13.10.3390/rs13224533 ↗
15. Garofalo, S.P., Ardito, F., Sanitate, N., De Carolis, G., Ruggieri, S., Giannico, V., Rana, G., and Ferrara, R.M. (2025). Robustness of Actual Evapotranspiration Predicted by Random Forest Model Integrating Remote Sensing and Meteorological Information: Case of Watermelon (Citrullus lanatus, (Thunb.) Matsum. & Nakai, 1916). Water, 17.10.3390/w17030323 ↗
16. Zhang, H.K., Qiu, S., Suh, J.W., Luo, D., and Zhu, Z. (2024). Machine learning and deep learning in remote sensing data analysis. Ref. Modul. Earth Syst. Environ. Sci.10.1016/B978-0-443-13220-9.00008-1 ↗
17. Pu, W., Wang, Z., Liu, D., and Zhang, Q. (2022). Optical Remote Sensing Image Cloud Detection with Self-Attention and Spatial Pyramid Pooling Fusion. Remote Sens., 14.10.3390/rs14174312 ↗
18. Multi-scale strip pooling feature aggregation network for cloud and cloud shadow segmentation10.1007/s00521-021-06802-0 ↗
19. Cloud/shadow segmentation based on global attention feature fusion residual network for remote sensing imagery10.1080/01431161.2020.1849852 ↗
20. ClouDet: A Dilated Separable CNN-Based Cloud Detection Framework for Remote Sensing Imagery10.1109/JSTARS.2021.3114171 ↗
21. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., and Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
22. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst. arXiv.
23. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
24. Liu, N., Zhang, N., Wan, K., Shao, L., and Han, J. (2021, January 10–17). Visual saliency transformer. Proceedings of the 18th IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.
25. Qiu, Y., Liu, Y., Zhang, L., and Xu, J. (2021). Boosting Salient Object Detection with Transformer-based Asymmetric Bilateral U-Net. arXiv.
26. Zhang, Y., Liu, H., and Hu, Q. (October, January 27). Transfuse: Fusing transformers and cnns for medical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Strasbourg, France.
27. Cloud-Graph: A feature interaction graph convolutional network for remote sensing image cloud detection
28. Gated aggregation network for cloud detection in remote sensing image10.1007/s00371-023-02934-7 ↗
29. Samplawski, C., and Marlin, B.M. (December, January 29). Towards Transformer-Based Real-Time Object Detection at the Edge: A Benchmarking Study. Proceedings of the MILCOM 2021-2021 IEEE Military Communications Conference, San Diego, CA, USA.10.1109/MILCOM52596.2021.9653052 ↗
30. Sun, Z., Cao, S., Yang, Y., and Kitani, K.M. (2021, January 11–17). Rethinking transformer-based set prediction for object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00359 ↗
31. Salient object detection in the deep learning era: An in-depth survey10.1109/TPAMI.2021.3051099 ↗
32. DefectTR: End-to-end defect detection for sewage networks using a transformer10.1016/j.conbuildmat.2022.126584 ↗
33. Botach, A., Zheltonozhskii, E., and Baskin, C. (2022, January 19–24). End-to-end referring video object segmentation with multimodal transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00493 ↗
34. Hu, J., Shen, L., and Sun, G. (2018, January 18–22). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
35. A tutorial on the cross-entropy method10.1007/s10479-005-5724-z ↗
36. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2019, January 16–20). Dual attention network for scene segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00326 ↗
37. Achanta, R., Hemami, S., Estrada, F., and Susstrunk, S. (2009, January 20–25). Frequency-tuned salient region detection. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPRW.2009.5206596 ↗
38. Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., and Shao, L. (2021, January 11–17). Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00061 ↗
39. Automated detection of cloud and cloud shadow in single-date Landsat imagery using neural networks and spatial post-processing10.3390/rs6064907 ↗
40. Siamese network for RGB-D salient object detection and beyond10.1109/TPAMI.2021.3073689 ↗
41. Margolin, R., Zelnik-Manor, L., and Tal, A. (2014, January 23–28). How to evaluate foreground maps?. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.39 ↗
42. Fan, D.P., Cheng, M.M., Liu, Y., Li, T., and Borji, A. (2017, January 22–29). Structure-measure: A new way to evaluate foreground maps. Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.487 ↗
43. Fan, D.P., Gong, C., Cao, Y., Ren, B., Cheng, M.M., and Borji, A. (2018). Enhanced-alignment measure for binary foreground map evaluation. arXiv.10.24963/ijcai.2018/97 ↗
44. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
45. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
46. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid scene parsing network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
47. Segnet: A deep convolutional encoder-decoder architecture for image segmentation10.1109/TPAMI.2016.2644615 ↗
48. Amirul Islam, M., Rochan, M., Bruce, N.D., and Wang, Y. (2017, January 21–26). Gated feedback refinement network for dense image labeling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.518 ↗
49. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., and Girdhar, R. (2022, January 18–24). Masked-attention Mask Transformer for Universal Image Segmentation. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00135 ↗
50. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
51. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
52. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗