Text-Guided Refinement for Referring Image Segmentation
Source abstract
Abstract
Referring image segmentation aims to segment an object described by a natural language expression from an image. Existing methods perform multi-modal fusion during encoding, typically integrating image and text features before predicting masks via upsampling networks. However, this approach often lacks sufficient multi-modal interactions during decoding, leading to challenges in achieving precise edge predictions for objects with varying scales. Additionally, the isolated interaction between linguistic and visual features at different scales fails to utilize the continuous guidance of language to multi-scale visual features. To address this issue, we propose the Text-Guided Refinement Network (TGRN). It employs a cascaded pyramid structure with a text-guided gating mechanism to enable selective and efficient integration of multi-modal features across multiple scales at the decoding stage. The proposed TGRN offers the following advantages: (a) It enhances information flow across feature scales, improving the network’s capacity to represent multi-scale semantics and achieve accurate segmentation. (b) It leverages text information to guide feature fusion, allowing for strengthened multi-modal interactions and refined edge perception during decoding. (c) It facilitates effective multi-modal information integration through a language-embedded visual encoder. Extensive experiments on three benchmark datasets validate the effectiveness of the proposed approach, demonstrating its superior performance in referring segmentation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Image Retrieval and Classification Techniques · Topic Modeling · Biomedical Text Mining and Ontologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Artificial Intelligence, Taiyuan University of Technology, Taiyuan 030024, China
School of Control and Computer Engineering, North China Electric Power University, Baoding 071003, China
The State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University, Beijing 100044, China
Funding
Fundamental Research Program of Shanxi Province · National Natural Science Foundation of China
References (55)
Show source-provided references
1. Hu, R., Rohrbach, M., and Darrell, T. (2016, January 11–14). Segmentation from natural language expressions. Proceedings of the ECCV, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_7 ↗
2. Li, R., Li, K., Kuo, Y.C., Shu, M., Qi, X., Shen, X., and Jia, J. (2018, January 18–22). Referring Image Segmentation via Recurrent Refinement Networks. Proceedings of the CVPR, Salt Lake City, UT, USA.10.1109/CVPR.2018.00602 ↗
3. Liu, C., Lin, Z., Shen, X., Yang, J., Lu, X., and Yuille, A. (2017). Recurrent Multimodal Interaction for Referring Image Segmentation. arXiv.10.1109/ICCV.2017.143 ↗
4. Margffoy-Tuay, E., Pérez, J.C., Botero, E., and Arbeláez, P. (2018). Dynamic Multimodal Instance Segmentation guided by natural language queries. arXiv.10.1007/978-3-030-01252-6_39 ↗
5. Shi, H., Li, H., Meng, F., and Wu, Q. (2018, January 8–14). Key-Word-Aware Network for Referring Expression Image Segmentation. Proceedings of the ECCV, Munich, Germany.10.1007/978-3-030-01231-1_3 ↗
6. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018). Encoder-decoder with atrous separable convolution for semantic image segmentation. arXiv.10.1007/978-3-030-01234-2_49 ↗
7. Lin, G., Shen, C., Van Den Hengel, A., and Reid, I. (July, January 26). Efficient piecewise training of deep structured models for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
8. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the CVPR, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
9. Scene segmentation with dag-recurrent neural networks10.1109/TPAMI.2017.2712691 ↗
10. Hu, Z., Feng, G., Sun, J., Zhang, L., and Lu, H. (2020, January 14–19). Bi-directional relationship inferring network for referring image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, DC, USA.10.1109/CVPR42600.2020.00448 ↗
11. Hui, T., Liu, S., Huang, S., Li, G., Yu, S., Zhang, F., and Han, J. (2020). Linguistic structure guided context modeling for referring image segmentation. Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Springer. Proceedings, Part X.10.1007/978-3-030-58607-2_4 ↗
12. Ye, L., Rochan, M., Liu, Z., and Wang, Y. (2019, January 16–20). Cross-modal self-attention network for referring image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Los Angeles, CA, USA.10.1109/CVPR.2019.01075 ↗
13. Yang, Z., Wang, J., Tang, Y., Chen, K., Zhao, H., and Torr, P.H. (2022, January 19–24). Lavt: Language-aware vision transformer for referring image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01762 ↗
14. VLT: Vision-language transformer and query generation for referring segmentation10.1109/TPAMI.2022.3217852 ↗
15. Wang, Z., Lu, Y., Li, Q., Tao, X., Guo, Y., Gong, M., and Liu, T. (2022, January 19–24). Cris: Clip-driven referring image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01139 ↗
16. Kim, N., Kim, D., Lan, C., Zeng, W., and Kwak, S. (2022, January 19–24). Restr: Convolution-free referring image segmentation using transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01761 ↗
17. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Virtual.
18. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023, January 2–6). Segment anything. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00371 ↗
19. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
20. Wei, Y., Xiao, H., Shi, H., Jie, Z., Feng, J., and Huang, T.S. (2018, January 18–22). Revisiting Dilated Convolution: A Simple Approach for Weakly-and Semi-Supervised Semantic Segmentation. Proceedings of the CVPR, Salt Lake City, UT, USA.10.1109/CVPR.2018.00759 ↗
21. Ding, H., Jiang, X., Shuai, B., Liu, A.Q., and Wang, G. (2018, January 18–22). Context contrasted feature and gated multi-scale aggregation for scene segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00254 ↗
22. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid scene parsing network. Proceedings of the CVPR, Hawaii, HI, USA.10.1109/CVPR.2017.660 ↗
23. Zhao, H., Zhang, Y., Liu, S., Shi, J., Loy, C.C., Lin, D., and Jia, J. (2018, January 8–14). Psanet: Point-wise spatial attention network for scene parsing. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01240-3_17 ↗
24. Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., and Liu, W. (November, January 27). Ccnet: Criss-cross attention for semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
25. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2019, January 16–20). Dual attention network for scene segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Los Angeles, CA, USA.10.1109/CVPR.2019.00326 ↗
26. Zhang, H., Zhang, H., Wang, C., and Xie, J. (2019, January 16–20). Co-occurrent features in semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Los Angeles, CA, USA.10.1109/CVPR.2019.00064 ↗
27. Rizzoli, G., Shenaj, D., and Zanuttigh, P. (2024, January 4–8). Source-free domain adaptation for rgb-d semantic segmentation with vision transformers. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Hawaii, HI, USA.10.1109/WACVW60836.2024.00070 ↗
28. Du, S., Wang, W., Guo, R., Wang, R., and Tang, S. (2024, January 17–21). Asymformer: Asymmetrical cross-modal representation learning for mobile platform real-time rgb-d semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, DC, USA.10.1109/CVPRW63382.2024.00756 ↗
29. Hu, R., Xu, H., Rohrbach, M., Feng, J., Saenko, K., and Darrell, T. (July, January 26). Natural language object retrieval. Proceedings of the CVPR, Las Vegas, NV, USA.
30. Liu, J., Wang, L., and Yang, M.H. (2017, January 22–29). Referring expression generation and comprehension via attributes. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.520 ↗
31. Zhao, P., Zheng, S., Zhao, W., Xu, D., Li, P., Cai, Y., and Huang, Q. (2024, January 20–27). Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i7.28580 ↗
32. LGR-NET: Language Guided Reasoning Network for Referring Expression Comprehension10.1109/TCSVT.2024.3374786 ↗
33. Liao, Y., Liu, S., Li, G., Wang, F., Chen, Y., Qian, C., and Li, B. (2020, January 14–19). A real-time cross-modality correlation filtering method for referring expression comprehension. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, DC, USA.10.1109/CVPR42600.2020.01089 ↗
34. Chen, D.J., Jia, S., Lo, Y.C., Chen, H.T., and Liu, T.L. (November, January 27). See-through-text grouping for referring image segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
35. Huang, S., Hui, T., Liu, S., Li, G., Wei, Y., Han, J., Liu, L., and Li, B. (2020, January 14–19). Referring image segmentation via cross-modal progressive comprehension. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, DC, USA.10.1109/CVPR42600.2020.01050 ↗
36. Chen, Y.W., Tsai, Y.H., Wang, T., Lin, Y.Y., and Yang, M.H. (2019). Referring expression object segmentation with caption-aware consistency. arXiv.
37. Referring image segmentation by generative adversarial learning10.1109/TMM.2019.2942480 ↗
38. Wang, H., Deng, C., Yan, J., and Tao, D. (November, January 27). Asymmetric cross-guided attention network for actor and action video segmentation from natural language query. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
39. Chen, Y., Wang, Z., Peng, Y., Zhang, Z., Yu, G., and Sun, J. (2018, January 18–22). Cascaded pyramid network for multi-person pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00742 ↗
40. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18–22). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00474 ↗
41. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
42. Yu, L., Poirson, P., Yang, S., Berg, A.C., and Berg, T.L. (2016, January 10–16). Modeling context in referring expressions. Proceedings of the ECCV, Amsterdam, The Netherlands.10.1007/978-3-319-46475-6_5 ↗
43. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 5–12). Microsoft coco: Common objects in context. Proceedings of the ECCV, Zurich, Switzerlan.10.1007/978-3-319-10602-1_48 ↗
44. The pascal visual object classes (voc) challenge10.1007/s11263-009-0275-4 ↗
45. Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., and Berg, T.L. (2018, January 18–22). Mattnet: Modular attention network for referring expression comprehension. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00142 ↗
46. Cross-Modal Progressive Comprehension for Referring Segmentation
47. Feng, G., Hu, Z., Zhang, L., and Lu, H. (2021, January 19–25). Encoder fusion network with co-attention embedding for referring image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.01525 ↗
48. Qiu, S., and Wang, W. (2023, January 10–12). Referring Image Segmentation via Text Guided Multi-Level Interaction. Proceedings of the 2023 IEEE/CIC International Conference on Communications in China (ICCC), Dalian, China.10.1109/ICCC57788.2023.10233585 ↗
49. Text-Vision Relationship Alignment for Referring Image Segmentation10.1007/s11063-024-11487-2 ↗
50. Jing, Y., Kong, T., Wang, W., Wang, L., Li, L., and Tan, T. (2021, January 19–25). Locate then segment: A strong pipeline for referring image segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.00973 ↗
51. Redmon, J. (July, January 26). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
52. Luo, G., Zhou, Y., Sun, X., Cao, L., Wu, C., Deng, C., and Ji, R. (2020, January 14–19). Multi-task collaborative network for joint referring expression comprehension and segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, DC, USA.10.1109/CVPR42600.2020.01005 ↗
53. Instance-specific feature propagation for referring segmentation10.1109/TMM.2022.3163578 ↗
54. Huang, Z., Xue, M., Liu, Y., Xu, K., Li, J., and Yu, C. (2024, January 3–6). DCMFNet: Deep Cross-Modal Fusion Network for Different Modalities with Iterative Gated Fusion. Proceedings of the 50th Graphics Interface Conference, Halifax, NS, Canada.10.1145/3670947.3670956 ↗
55. Multi-modal mutual attention and iterative interaction for referring image segmentation10.1109/TIP.2023.3277791 ↗