OpenMamba: Introducing State Space Models to Open-Vocabulary Semantic Segmentation
Source abstract
Abstract
Open-vocabulary semantic segmentation aims to label each pixel of an image based on text descriptions provided at inference time. Recent approaches for this task are based on methods which require two stages: the first one uses a mask generator to generate mask proposals, while the other one deals with segment classification using a pre-trained vision–language model, such as CLIP. However, since CLIP is pre-trained on natural images, the model struggles with segmentation masks because of their abstract nature. In this paper, we introduce OpenMamba, a novel approach to creating high-level guidance maps to assist in extracting CLIP features within the masked regions for classification. High-level guidance maps are generated by leveraging both visual and textual modalities and introducing State Space Duality (SSD) as an efficient way to tackle the open-vocabulary semantic segmentation task. Also, we propose a new matching technique for the mask proposals, based on IoU with a dynamic threshold conditioned by mask quality and we introduce a contrastive-based loss to assure that similar mask proposals achieve similar CLIP embeddings. Comprehensive experiments across open-vocabulary benchmarks show that our method can achieve superior performance compared to other approaches while managing to reduce memory consumption.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computers and Information Technology, Politehnica University of Timișoara, 300223 Timișoara, Romania
Funding
Politehnica University of Timișoara
References (58)
Show source-provided references
1. Štarchoň, P., Fedushko, S., and Gubíniová, K. (2024). Image Processing Application Development: A New Approach and Its Economic Profitability. Data-Centric Business and Applications, Springer Nature. Lecture Notes on Data Engineering and Communications Technologies.10.1007/978-3-031-60815-5 ↗
2. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
3. Strudel, R., Garcia, R., Laptev, I., and Schmid, C. (2021, January 10–17). Segmenter: Transformer for semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00717 ↗
4. Segformer: Simple and efficient design for semantic segmentation with transformers
5. Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., and Torr, P.H.S. (2021, January 20–25). Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00681 ↗
6. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Virtual.
7. Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. (2021, January 18–24). Scaling up visual and vision-language representation learning with noisy text supervision. Proceedings of the International Conference on Machine Learning, Virtual.
8. Cho, S., Shin, H., Hong, S., Arnab, A., Seo, P.H., and Kim, S. (2024, January 16–22). Catseg: Cost aggregation for open-vocabulary semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00394 ↗
9. Zhou, Z., Lei, Y., Zhang, B., Liu, L., and Liu, Y. (2022). Zegclip: Towards adapting clip for zero-shot semantic segmentation. arXiv.10.1109/CVPR52729.2023.01075 ↗
10. Xu, M., Zhang, Z., Wei, F., Lin, Y., Cao, Y., Hu, H., and Bai, X. (2022, January 23–27). A simple baseline for open vocabulary semantic segmentation with pre-trained vision language model. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19818-2_42 ↗
11. Jiao, S., Zhu, H., Huang, J., Zhao, Y., Wei, Y., and Shi, H. (2024). Collaborative vision-text representation optimizing for open-vocabulary segmentation. arXiv.10.1007/978-3-031-73414-4_23 ↗
12. Shan, X., Wu, D., Zhu, G., Shao, Y., Sang, N., and Gao, C. (2024, January 16–22). Open-vocabulary semantic segmentation with image embedding balancing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02684 ↗
13. Ding, J., Xue, N., Xia, G.S., and Dai, D. (2022, January 18–24). Decoupling zero-shot semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01129 ↗
14. Han, C., Zhong, Y., Li, D., Han, K., and Ma, L. (2023, January 1–6). Open-vocabulary semantic segmentation with decoupled one-pass network. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00106 ↗
15. Li, Y., Cheng, T., Feng, B., Liu, W., and Wang, X. (2024). Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation. arXiv.10.1109/CVPR52734.2025.01397 ↗
16. Shi, Y., Dong, M., Li, M., and Xu, C. (2024). VSSD: Vision Mamba with Non-Causal State Space Duality. arXiv.
17. Oord, A., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv.
18. Zhang, X., Chi, C., Jinnai, Y., Li, Y., Zhang, X., Wei, Y., and Sun, J. (2020, January 11–15). Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00978 ↗
19. Semantic understanding of scenes through the ade20k dataset
20. The pascal visual object classes (voc) challenge10.1007/s11263-009-0275-4 ↗
21. Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., and Yuille, A. (2014, January 23–28). The role of context for object detection and semantic segmentation in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.119 ↗
22. Chen, Y.-C., Li, L., Yu, L., Kholy, A.E., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. (2020, January 23–28). UNITER: UNiversal Image-TExt Representation Learning. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XXX 30.10.1007/978-3-030-58577-8_7 ↗
23. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., and Dai, J. (2019). Vl-bert: Pre-training of generic visual-linguistic representations. arXiv.
24. Lu, J., Batra, D., Parikh, D., and Lee, S. (2019, January 8–14). Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Proceedings of the Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Vancouver, BC, Canada.
25. Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., and Wei, F. (2020, January 23–28). Oscar: Object-semantics aligned pre-training for vision-language tasks. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part XXX 16.10.1007/978-3-030-58577-8_8 ↗
26. Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L.H., Zhou, L., Dai, X., Yuan, L., and Li, Y. (2022, January 18–24). Regionclip: Region based language-image pretraining. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01629 ↗
27. Bucher, M., Vu, T.H., Cord, M., and Perez, P. (2019, January 8–14). Zero-shot semantic segmentation. Proceedings of the Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Vancouver, BC, Canada.
28. Xu, M., Zhang, Z., Wei, F., Hu, H., and Bai, X. (2023, January 17–24). Side adapter network for open-vocabulary semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00288 ↗
29. Yu, Q., He, J., Deng, X., Shen, X., and Chen, L.C. (2023, January 10–16). Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Proceedings of the Advances in Neural Information Processing Systems 36 (NeurIPS 2023), New Orleans, LA, USA.
30. Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. (2021, January 10–17). Styleclip: Text-driven manipulation of stylegan imagery. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00209 ↗
31. Ghiasi, G., Gu, X., Cui, Y., and Lin, T.Y. (2022, January 23–27). Scaling open-vocabulary image segmentation with image-level labels. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20059-5_31 ↗
32. Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., and Marculescu, D. (2023, January 17–24). Open-vocabulary semantic segmentation with mask-adapted clip. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00682 ↗
33. Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., and Mello, S.D. (2023, January 17–24). Open-vocabulary panoptic segmentation with text-to-image diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00289 ↗
34. Wang, X., He, W., Xuan, X., Sebastian, C., Ono, J.P., Li, X., Behpour, S., Doan, T., Gou, L., and Shen, H.W. (2024). USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation. arXiv.10.1109/CVPR52733.2024.00401 ↗
35. Chng, Y.X., Qiu, X., Han, Y., Ding, K., Ding, W., and Huang, G. (2024). Adapting Vision-Language Model with Fine-grained Semantics for Open-Vocabulary Segmentation. arXiv.
36. Ding, Z., Wang, J., and Tu, Z. (2022). Open vocabulary universal image segmentation with maskclip. arXiv.
37. Xie, B., Cao, J., Xie, J., Khan, F.S., and Pang, Y. (2024, January 16–22). Sed: A simple encoder-decoder for open vocabulary semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00329 ↗
38. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.-Y. (2023, January 2–3). Segment anything. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00371 ↗
39. Zhu, Y., Zhu, B., Chen, Z., Xu, H., Tang, M., and Wang, J. (2024). Mrovseg: Breaking the resolution curse of vision-language models in open-vocabulary semantic segmentation. arXiv.
40. Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and Ré, C. (2021, January 6–14). Combining Recurrent, Convolutional, and Continuous-time Models with Linear StateSpace Layers. Proceedings of the Advances in Neural Information Processing Systems 34 (NeurIPS 2021), Online.
41. Gu, A., Dao, T., Ermon, S., Rudra, A., and Ré, C. (2020, January 6–12). HiPPO: Recurrent Memory with Optimal Polynomial Projections. Proceedings of the Advances in Neural Information Processing Systems 33 (NeurIPS 2020), Virtual.
42. Gu, A., and Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv.
43. Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., and Wang, X. (2024). Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv.
44. Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y. (2024). Vmamba: Visual state space model. arXiv.
45. Ma, J., Li, F., and Wang, B. (2024). U-mamba: Enhancing longrange dependency for biomedical image segmentation. arXiv.
46. Liang, D., Zhou, X., Wang, X., Zhu, X., Xu, W., Zou, Z., Ye, X., and Bai, X. (2024). Pointmamba: A simple state space model for point cloud analysis. arXiv.
47. Zhang, T., Li, X., Yuan, H., Ji, S., and Yan, S. (2024). Point could mamba: Point cloud learning via state space model. arXiv.10.1609/aaai.v39i10.33098 ↗
48. Lin, B., Jiang, W., Chen, P., Zhang, Y., Liu, S., and Chen, Y.-C. (October, January 29). MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoders. Proceedings of the European Conference on Computer Vision (ECCV), Milan, Italy.
49. Dao, T., and Gu, A. (2024). Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv.
50. Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., and Girdhar, R. (2022, January 18–24). Masked-attention mask transformer for universal image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00135 ↗
51. Caesar, H., Uijlings, J., and Ferrari, V. (2018, January 18–23). Cocostuff: Thing and stuff classes in context. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00132 ↗
52. Loshchilov, I. (2017). Decoupled weight decay regularization. arXiv.
53. Xu, J., Mello, S.D., Liu, S., Byeon, W., Breuel, T., Kautz, J., and Wang, X. (2022, January 18–24). GroupViT: Semantic segmentation emerges from text supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01760 ↗
54. Sharma, P., Ding, N., Goodman, S., and Soricut, R. (2018, January 15–20). Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.10.18653/v1/P18-1238 ↗
55. YFCC100M: The new data in multimedia research10.1145/2812802 ↗
56. Kirillov, A., He, K., Girshick, R., Rother, C., and Dollar, P. (2019, January 15–20). Panoptic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00963 ↗
57. Liu, Y., Bai, S., Li, G., Wang, Y., and Tang, Y. (2024, January 16–22). Open-Vocabulary Segmentation with Semantic-Assisted Calibration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.00335 ↗
58. Peng, Z., Xu, Z., Zeng, Z., Wang, Y., and Shen, W. (2024). Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation. arXiv.10.1109/CVPR52734.2025.01398 ↗