Structured Scene Parsing with a Hierarchical CLIP Model for Images
Source abstract
Abstract
Visual Relationship Prediction (VRP) is crucial for advancing structured scene understanding, yet existing methods struggle with ineffective multimodal fusion, static relationship representations, and a lack of logical consistency. To address these limitations, this paper proposes a Hierarchical CLIP model (H-CLIP) for structured scene parsing. Our approach leverages a pre-trained CLIP backbone to extract aligned visual, textual, and spatial features for entities and their union regions. A multi-head self-attention mechanism then performs deep, dynamic multimodal fusion. The core innovation is a consistency and reversibility verification mechanism, which imposes algebraic constraints as a regularization loss to enforce logical coherence in the learned relation space. Extensive experiments on the Visual Genome dataset demonstrate the superiority of the proposed method. H-CLIP significantly outperforms state-of-the-art baselines on the predicate classification task, achieving a Recall@50 score of 64.31% and a Mean Recall@50 of 36.02%, thereby validating its effectiveness in generating accurate and logically consistent scene graphs even under long-tailed distributions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Software, Dalian University of Foreign Languages, Dalian 116044, China
Research Center for Language Intelligence, Dalian University of Foreign Languages, Dalian 116044, China
References (23)
Show source-provided references
1. Neau, M., Santos, P., Bosser, A.G., and Buche, C. (2023). In Defense of Scene Graph Generation for Human-Robot Open-Ended Interaction in Service Robotics. Robot World Cup, Springer.10.1007/978-3-031-55015-7_25 ↗
2. Girshick, R. (2015, January 7–13). Fast R-CNN. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.169 ↗
3. Dai, J., He, K., and Sun, J. (2016, January 27–30). Instance-Aware Semantic Segmentation via Multi-Task Network Cascades. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.343 ↗
4. Zareian, A., Karaman, S., and Chang, S.-F. (2020, January 13–19). Weakly Supervised Visual Semantic Parsing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00379 ↗
5. Zhang, H., Kyaw, Z., Chang, S.-F., and Chua, T.-S. (2017, January 21–26). Visual Translation Embedding Network for Visual Relation Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.331 ↗
6. Zellers, R., Yatskar, M., Thomson, S., and Choi, Y. (2018, January 18–22). Neural Motifs: Scene Graph Parsing with Global Context. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00611 ↗
7. Tang, K., Zhang, H., Wu, B., Luo, W., and Liu, W. (2019, January 15–20). Learning to Compose Dynamic Tree Structures for Visual Contexts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00678 ↗
8. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
9. Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D., Bernstein, M., and Li, F.-F. (2015, January 7–12). Image Retrieval Using Scene Graphs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298990 ↗
10. Xu, D., Zhu, Y., Choy, C.B., and Li, F.-F. (2017, January 21–26). Scene Graph Generation by Iterative Message Passing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.330 ↗
11. Li, Y., Ouyang, W., Zhou, B., Wang, K., and Wang, X. (2017, January 21–26). VIP-CNN: Visual Phrase Guided Convolutional Neural Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.766 ↗
12. Lu, C., Krishna, R., Bernstein, M., and Li, F.-F. (2016, January 11–14). Visual Relationship Detection with Language Priors. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_51 ↗
13. Mi, L., and Chen, Z. (2020, January 13–19). Hierarchical Graph Attention Network for Visual Relationship Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01390 ↗
14. Reltr: Relation Transformer for Scene Graph Generation10.1109/TPAMI.2023.3268066 ↗
15. Translating Embeddings for Modeling Multi-Relational Data
16. Contextual Translation Embedding for Visual Relationship Detection and Scene Graph Generation10.1109/TPAMI.2020.2992222 ↗
17. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the International Conference on Machine Learning (ICML), Virtual. Proceedings of Machine Learning Research.
18. Subramanyam, R., Jayram, T.S., Anirudh, R., and Thiagarajan, J.J. (2024, January 14–19). Exploring the Utility of Clip Priors for Visual Relationship Prediction. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10447556 ↗
19. Mitra, C., Huang, B., Darrell, T., and Rohrbach, A. (2024, January 16–22). Compositional Chain-of-Thought Prompting for Large Multimodal Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01367 ↗
20. Yu, Q., Li, J., Wu, Y., Zhang, Y., Wang, Y., Wang, Y., and Yang, M. (2023, January 2–6). Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.01971 ↗
21. Jiang, B., Liu, X., Wang, Y., and Taylor, C.J. (2025, January 3–8). Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV61041.2025.00861 ↗
22. Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, J. (2022). When and Why Vision-Language Models Behave Like Bags-of-Words, and What to Do About It?. arXiv.
23. Zhao, Y., Shu, T., Katz, B., Barbu, A., and Mao, J. (2022). VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations. arXiv.10.18653/v1/2022.emnlp-demos.4 ↗