Few-Shot Semantic Segmentation Network for Distinguishing Positive and Negative Examples
Source abstract
Abstract
Few-shot segmentation (FSS) aims to segment a query image with a few support images. However, there can be large differences between images from the same category, and similarities between different categories, making it a challenging task. In addition, most FSS methods use powerful encoders to extract features from the training class, which makes the model pay more attention to the features of the ‘seen’ class, and perform poorly on the segmentation task of ‘unseen’ classes. In this work, we propose a novel end-to-end model, called GFormer. GFormer has four components: encoder, prototype extractor, adversarial prototype generator, and decoder. Our encoder makes simple modifications to VIT to reduce the focus on image content, using a prototype extractor to extract prototype features from a range of support images. We further introduce different classes that are similar to the support image categories as negative examples, taking the support image categories as positive examples. We use the adversarial prototype generator to extract the adversarial prototypes from the positive and negative examples. The decoder segments the query images under the guidance of the prototypes. We conduct extensive experiments on a variety of unknown classes. The results verify the feasibility of the proposed model and prove that the proposed model has strong generalization performance for new classes.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Adversarial Robustness in Machine Learning · Anomaly Detection Techniques and Applications · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Research Institute of Electronic Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, China
References (30)
Show source-provided references
1. Nguyen, K., and Todorovic, S. (November, January 27). Feature weighting and boosting for few-shot segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
2. Shaban, A., Bansal, S., Liu, Z., Essa, I., and Boots, B. (2017). One-shot learning for semantic segmentation. arXiv.10.5244/C.31.167 ↗
3. Sg-one: Similarity guidance network for one-shot semantic segmentation10.1109/TCYB.2020.2992433 ↗
4. Wang, K., Liew, J.H., Zou, Y., Zhou, D., and Feng, J. (November, January 27). Panet: Few-shot image semantic segmentation with prototype alignment. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
5. Liu, J., and Qin, Y. (2020). Prototype refinement network for few-shot segmentation. arXiv.
6. Mask matching transformer for few-shot segmentation
7. Wang, Y., Sun, R., Zhang, Z., and Zhang, T. (2022, January 23–27). Adaptive agent transformer for few-shot segmentation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19818-2_3 ↗
8. Strudel, R., Garcia, R., Laptev, I., and Schmid, C. (2021, January 11–17). Segmenter: Transformer for semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00717 ↗
9. Generative adversarial networks10.1145/3422622 ↗
10. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, C.A., and Lo, W.-Y. (2023, January 2–3). Segment anything. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00371 ↗
11. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An image is worth 16x16 words: Transformers for image recognition at scale. Proceedings of the 9th International Conference on Learning Representations, ICLR 2021, Virtual.
12. Ronneberger, O., Philipp, F., and Thomas, B. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany. Part III 18.
13. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2014). Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv.
14. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
15. Chen, L.C., Papandreou, G., Schroff, F., and Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. arXiv.
16. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
17. Per-pixel classification is not all you need for semantic segmentation
18. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., and Girdhar, R. (2022, January 18–24). Masked-attention mask transformer for universal image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00135 ↗
19. SegFormer: Simple and efficient design for semantic segmentation with transformers
20. Few-shot semantic segmentation with prototype learning
21. Siam, M., Oreshkin, B.N., and Jagersand, M. (November, January 27). Amp: Adaptive masked proxies for few-shot segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
22. Vinyals, O., Blundell, C., Lillicrap, T., and Wierstra, D. (2016). Matching networks for one shot learning. Adv. Neural Inf. Process. Syst., 29.
23. Vaswani, A. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
24. Wu, K., Peng, H., Chen, M., Fu, J., and Chao, H. (2021, January 11–17). Rethinking and improving relative position encoding for vision transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00988 ↗
25. Li, X., Wei, T., Chen, Y.P., Tai, Y.W., and Tang, C.K. (2020, January 14–19). Fss-1000: A 1000-class dataset for few-shot segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00294 ↗
26. The pascal visual object classes (voc) challenge10.1007/s11263-009-0275-4 ↗
27. Rakelly, K., Shelhamer, E., Darrell, T., Efros, A.A., and Levine, S. (2018). Few-shot segmentation propagation with guided networks. arXiv.
28. Hendryx, S.M., Leach, A.B., Hein, P.D., and Morrison, C.T. (2019). Meta-learning initializations for image segmentation. arXiv.
29. Zhang, C., Lin, G., Liu, F., Yao, R., and Shen, C. (2019, January 15–20). Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00536 ↗
30. Lu, Z., He, S., Zhu, X., Zhang, L., Song, Y.Z., and Xiang, T. (, January 11–17). Simpler is better: Few-shot semantic segmentation with classifier weight transformer. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.