Enhanced Pseudo-Labels for End-to-End Weakly Supervised Semantic Segmentation with Foundation Models
Source abstract
Abstract
Weakly supervised semantic segmentation (WSSS) aims to learn pixel-level semantic concepts from image-level class labels. Due to the simplicity and efficiency of the training pipeline, end-to-end WSSS has received significant attention from the research community. However, the coarse nature of pseudo-label regions remains one of the primary bottlenecks limiting the performance of such methods. To address this issue, we propose class-guided enhanced pseudo-labeling (CEP), a method designed to generate high-quality pseudo-labels for end-to-end WSSS frameworks. Our approach leverages pretrained foundation models, such as contrastive language-image pre-training (CLIP) and segment anything model (SAM), to enhance pseudo-label quality. Specifically, following the pseudo-label generation pipeline, we introduce two key components: a Skip-CAM module and a pseudo-label refinement module. The Skip-CAM module enriches feature representations by introducing skip connections from multiple blocks of CLIP, thereby improving the quality of localization maps. The refinement module then utilizes SAM to refine and correct the pseudo-labels based on the initial class-specific regions derived from the localization maps. Experimental results demonstrate that our method surpasses the state-of-the-art end-to-end approaches as well as many multi-stage competitors.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Machine Learning and Data Classification · Handwritten Text Recognition Techniques · Text and Document Classification Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer and Information Engineering, Fuyang Normal University, Fuyang 236037, China
Wenzhou Key Laboratory for Intelligent Networking, Wenzhou University, Wenzhou 325035, China
Funding
Zhejiang Provincial Natural Science Foundation of China
References (58)
Show source-provided references
1. Ahn, J., Cho, S., and Kwak, S. (2019, January 16–20). Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00231 ↗
2. Yoon, S.H., Kwon, H., Kim, H., and Yoon, K.J. (2024, January 17–21). Class Tokens Infusion for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00345 ↗
3. Oh, Y., Kim, B., and Ham, B. (2021, January 16–20). Background-Aware Pooling and Noise-Aware Loss for Weakly-Supervised Semantic Segmentation. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual.10.1109/CVPR46437.2021.00684 ↗
4. Lee, J., Yi, J., Shin, C., and Yoon, S. (2021, January 16–20). BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual.10.1109/CVPR46437.2021.00267 ↗
5. Xu, J., Zhou, C., Cui, Z., Xu, C., Huang, Y., Shen, P., Li, S., and Yang, J. (2021, January 11–17). Scribble-Supervised Semantic Segmentation Inference. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Virtual.10.1109/ICCV48922.2021.01507 ↗
6. CC4S: Encouraging Certainty and Consistency in Scribble-Supervised Semantic Segmentation10.1109/TPAMI.2024.3415387 ↗
7. Ru, L., Zhan, Y., Yu, B., and Du, B. (2022, January 19–24). Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01634 ↗
8. Ru, L., Zheng, H., Zhan, Y., and Du, B. (2023, January 18–22). Token Contrast for Weakly-Supervised Semantic Segmentation. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00302 ↗
9. Zhang, B., Yu, S., Wei, Y., Zhao, Y., and Xiao, J. (2024, January 17–21). Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00364 ↗
10. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (July, January 26). Learning Deep Features for Discriminative Localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
11. Learning Transferable Visual Models From Natural Language Supervision
12. Lin, Y., Chen, M., Wang, W., Wu, B., Li, K., Lin, B., Liu, H., and He, X. (2023, January 18–22). CLIP Is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01469 ↗
13. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023, January 2–6). Segment Anything. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00371 ↗
14. Yang, X., and Gong, X. (2024, January 4–8). Foundation Model Assisted Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV57701.2024.00058 ↗
15. Kweon, H., and Yoon, K.J. (2024, January 17–21). From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01844 ↗
16. Dalal, N., and Triggs, B. (2005, January 20–25). Histograms of oriented gradients for human detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA.
17. Distinctive image features from scale-invariant keypoints10.1023/B:VISI.0000029664.99615.94 ↗
18. Fulkerson, B., Vedaldi, A., and Soatto, S. (2009, January 20–25). Class segmentation and object localization with superpixel neighborhoods. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Miami, FL, USA.10.1109/ICCV.2009.5459175 ↗
19. Shotton, J., Johnson, M., and Cipolla, R. (2008, January 23–28). Semantic texton forests for image categorization and segmentation. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.10.1109/CVPR.2008.4587503 ↗
20. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully Convolutional Networks for Semantic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
21. Liang-Chieh, C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. (2015, January 7–9). Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
22. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations, Virtual.
23. Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. (2021, January 11–17). Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction Without Convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Virtual.10.1109/ICCV48922.2021.00061 ↗
24. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., and Girdhar, R. (2022, January 19–24). Masked-Attention Mask Transformer for Universal Image Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00135 ↗
25. Ahn, J., and Kwak, S. (2018, January 18–22). Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00523 ↗
26. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
27. Chen, Z., Wang, T., Wu, X., Hua, X.S., Zhang, H., and Sun, Q. (2022, January 19–24). Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00104 ↗
28. Lee, M., Kim, D., and Shim, H. (2022, January 19–24). Threshold Matters in WSSS: Manipulating the Activation for the Robust and Accurate Segmentation Model Against Thresholds. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00429 ↗
29. Xu, L., Ouyang, W., Bennamoun, M., Boussaid, F., and Xu, D. (2022, January 19–24). Multi-class Token Transformer for Weakly Supervised Semantic Segmentation. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00427 ↗
30. Credible Dual-Expert Learning for Weakly Supervised Semantic Segmentation10.1007/s11263-023-01796-9 ↗
31. Zhang, B., Xiao, J., Wei, Y., Sun, M., and Huang, K. (2020, January 7–12). Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6971 ↗
32. Wider or Deeper: Revisiting the ResNet Model for Visual Recognition10.1016/j.patcog.2019.01.006 ↗
33. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017, January 4–9). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
34. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
35. Wu, Y., Ye, X., Yang, K., Li, J., and Li, X. (2024, January 17–21). DuPL: Dual Student with Trustworthy Progressive Learning for Robust Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00339 ↗
36. Xie, J., Hou, X., Ye, K., and Shen, L. (2022, January 19–24). CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.00444 ↗
37. WeakCLIP: Adapting CLIP for Weakly-Supervised Semantic Segmentation10.1007/s11263-024-02224-2 ↗
38. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22–29). Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.74 ↗
39. The Pascal Visual Object Classes (VOC) Challenge10.1007/s11263-009-0275-4 ↗
40. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft COCO: Common Objects in Context. Proceedings of the European Conference on Computer Vision (ECCV), Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
41. Hariharan, B., Arbeláez, P., Bourdev, L., Maji, S., and Malik, J. (2011, January 6–13). Semantic contours from inverse detectors. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.10.1109/ICCV.2011.6126343 ↗
42. Parameter Learning and Convergent Inference for Dense Random Fields
43. Yang, Z., Meng, Y., Fu, K., Wang, S., and Song, Z. (March, January 25). MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA.
44. Yang, Z., Zhao, X., Wang, X., Zhang, Q., and Xiao, J. (2025, January 11–15). FFR: Frequency Feature Rectification for Weakly Supervised Semantic Segmentation. Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), Nashville, TN, USA.10.1109/CVPR52734.2025.02817 ↗
45. Yang, Z., Fu, K., Duan, M., Qu, L., Wang, S., and Song, Z. (2024, January 17–21). Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00346 ↗
46. Wang, J., Dai, T., Zhang, B., Yu, S., Lim, E.G., and Xiao, J. (2025, January 11–15). POT: Prototypical Optimal Transport for Weakly Supervised Semantic Segmentation. Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), Nashville, TN, USA.10.1109/CVPR52734.2025.01402 ↗
47. Rong, S., Tu, B., Wang, Z., and Li, J. (2023, January 18–22). Boundary-Enhanced Co-Training for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01875 ↗
48. Kweon, H., Yoon, S.H., and Yoon, K.J. (2023, January 18–22). Weakly Supervised Semantic Segmentation via Adversarial Learning of Classifier and Reconstructor. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01090 ↗
49. Cheng, Z., Qiao, P., Li, K., Li, S., Wei, P., Ji, X., Yuan, L., Liu, C., and Chen, J. (2023, January 18–22). Out-of-Candidate Rectification for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.02267 ↗
50. Peng, Z., Wang, G., Xie, L., Jiang, D., Shen, W., and Tian, Q. (2023, January 2–6). USAGE: A Unified Seed Area Generation Paradigm for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.00064 ↗
51. Chen, Z., and Sun, Q. (2023, January 18–22). Extracting Class Activation Maps From Non-Discriminative Features As Well. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.00306 ↗
52. Yoon, S.H., Kwon, H., Jeong, J., Park, D., and Yoon, K.J. (October, January 29). Diffusion-Guided Weakly Supervised Semantic Segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy.
53. Duan, S., Yang, X., and Wang, N. (2025, January 11–15). Multi-Label Prototype Visual Spatial Search for Weakly Supervised Semantic Segmentation. Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), Nashville, TN, USA.10.1109/CVPR52734.2025.02815 ↗
54. Zhao, X., Yang, Z., Dai, T., Zhang, B., and Xiao, J. (2024, January 17–21). PSDPM: Prototype-based Secondary Discriminative Pixels Mining for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00330 ↗
55. Tang, F., Xu, Z., Qu, Z., Feng, W., Jiang, X., and Ge, Z. (2024, January 17–21). Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00320 ↗
56. Xu, R., Wang, C., Sun, J., Xu, S., Meng, W., and Zhang, X. (2023, January 7–14). Self Correspondence Distillation for End-to-End Weakly-Supervised Semantic Segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.10.1609/aaai.v37i3.25408 ↗
57. Araslanov, N., and Roth, S. (2020, January 14–19). Single-Stage Semantic Segmentation From Image Labels. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00431 ↗
58. Learning Self-supervised Low-Rank Network for Single-Stage Weakly and Semi-supervised Semantic Segmentation10.1007/s11263-022-01590-z ↗