journal article·2026-07-28

Global-to-Local Visual Conditioning for Image Captioning with a Frozen Vision–Language Model

Sohyun Lee, Daeyoung Heo

Applied Sciences· Volume 16· Issue 15
Publisher page

Abstract

Image captioning with pretrained vision and language models often requires substantial model adaptation, while lightweight settings restrict the number of components that can be updated. This study investigates an input-level visual conditioning approach for image captioning using a frozen ResNet-50 image encoder and a frozen Phi-3 Mini language model. A global image representation is first projected into ten continuous visual tokens. A gated local residual module is then introduced to incorporate spatial features from the final ResNet feature map without modifying either pretrained backbone. The framework was evaluated on the MS COCO Karpathy split under a fixed training and decoding protocol. The global visual-conditioning baseline achieved a CIDEr score of 0.8382 on the test split. Adding the gated local residual module increased CIDEr to 0.8693, while adding only 3.95 million stage-specific trainable parameters. Under a near-matched trainable-parameter budget, a rank-10 QKV-only LoRA control achieved a CIDEr score of 0.8439, whereas direct and gated local residual conditioning achieved 0.8611 and 0.8693, respectively. These results are limited to the evaluated frozen ResNet-50–Phi-3 configuration and do not imply a general advantage over LoRA variants. As an additional sequence-level optimization experiment, self-critical sequence training increased CIDEr marginally from 0.8693 to 0.8704 without changing the inference architecture. Caption quality was further examined using a structured LLM-based evaluation over all 5000 images in the Karpathy test split, assessing semantic accuracy, information coverage, fluency, and reference-based unsupported content. A zero-shot evaluation on Flickr30k showed a marked performance decrease, indicating limited cross-dataset generalization. Overall, the experiments provide an empirical analysis of global and local visual conditioning for parameter-efficient caption generation with frozen pretrained components.
Source: crossref · Quality: baseline

Affiliations

Department of Computer Science, Kookmin University, Seoul 08826, Republic of Korea

Funding

Ministry of the Interior and Safety

References (36)

Show source-provided references
1. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016, IEEE.10.1109/CVPR.2016.90 ↗
2. Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A.A., Bach, N., Bahree, A., Bakhtiari, A., Bao, H., and Behl, H. (2024). Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv.
3. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022, January 25–29). LoRA: Low-Rank Adaptation of Large Language Models. Proceedings of the International Conference on Learning Representations (ICLR), Virtual.
4. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., and Goel, V. (2017). Self-Critical Sequence Training for Image Captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017, IEEE.10.1109/CVPR.2017.131 ↗
5. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014). Microsoft COCO: Common Objects in Context. Computer Vision—ECCV 2014, Proceedings of the 13th European Conference, Zurich, Switzerland, 6–12 September 2014, Springer.10.1007/978-3-319-10602-1_48 ↗
6. Karpathy, A., and Fei-Fei, L. (2015). Deep Visual-Semantic Alignments for Generating Image Descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015, IEEE.10.1109/CVPR.2015.7298932 ↗
7. From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference over Event Descriptions10.1162/tacl_a_00166 ↗
8. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015). Show and Tell: A Neural Image Caption Generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015, IEEE.10.1109/CVPR.2015.7298935 ↗
9. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
10. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30, Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017, Curran Associates, Inc.
11. Cornia, M., Stefanini, M., Baraldi, L., and Cucchiara, R. (2020). Meshed-Memory Transformer for Image Captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020, IEEE.10.1109/CVPR42600.2020.01059 ↗
12. BLIP: Bootstrapping Language–Image Pre-Training for Unified Vision–Language Understanding and Generation
13. BLIP-2: Bootstrapping Language–Image Pre-Training with Frozen Image Encoders and Large Language Models
14. Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. (2023). InstructBLIP: Towards General-Purpose Vision–Language Models with Instruction Tuning. Advances in Neural Information Processing Systems 36, Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NIPS 2023), New Orleans, LA, USA, 10–16 December 2023, Curran Associates, Inc.10.52202/075280-2142 ↗
15. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., and Reynolds, M. (2022). Flamingo: A Visual Language Model for Few-Shot Learning. Advances in Neural Information Processing Systems 35, Proceedings of the 36th Annual Conference on Neural Information Processing Systems (NIPS 2022), New Orleans, LA, USA, 28 November–9 December 2022, Curran Associates, Inc.10.52202/068431-1723 ↗
16. Liu, H., Li, C., Wu, Q., and Lee, Y.J. (2023). Visual Instruction Tuning. Advances in Neural Information Processing Systems 36, Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NIPS 2023), New Orleans, LA, USA, 10–16 December 2023, Curran Associates, Inc.10.52202/075280-1516 ↗
17. Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. (2024, January 7–11). Kosmos-2: Grounding Multimodal Large Language Models to the World. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria.
18. Tsimpoukelli, M., Menick, J.L., Cabi, S., Eslami, S.M.A., Vinyals, O., and Hill, F. (2021). Multimodal Few-Shot Learning with Frozen Language Models. Advances in Neural Information Processing Systems 34, Proceedings of the 35th Annual Conference on Neural Information Processing Systems (NIPS 2021), Virtual, 6–14 December 2021, Curran Associates, Inc.
19. Parameter-Efficient Transfer Learning for NLP
20. Li, X.L., and Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Online, 1–6 August 2021, Association for Computational Linguistics.10.18653/v1/2021.acl-long.353 ↗
21. Lester, B., Al-Rfou, R., and Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, 7–11 November 2021, Association for Computational Linguistics.10.18653/v1/2021.emnlp-main.243 ↗
22. Mokady, R., Hertz, A., and Bermano, A.H. (2021). ClipCap: CLIP Prefix for Image Captioning. arXiv.
23. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA, USA, 7–12 July 2002, Association for Computational Linguistics.10.3115/1073083.1073135 ↗
24. Banerjee, S., and Lavie, A. (2005). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA, 29 June 2005, Association for Computational Linguistics.
25. Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, Proceedings of the ACL-04 Workshop, Barcelona, Spain, 21–26 July 2004, Association for Computational Linguistics.
26. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015). CIDEr: Consensus-Based Image Description Evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015, IEEE.10.1109/CVPR.2015.7299087 ↗
27. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., and Xing, E.P. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36, Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NeurIPS 2023), New Orleans, LA, USA, 10–16 December 2023, Curran Associates, Inc.10.52202/075280-2020 ↗
28. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore, 6–10 December 2023, Association for Computational Linguistics.10.18653/v1/2023.emnlp-main.153 ↗
29. Chan, D.M., Petryk, S., Gonzalez, J.E., Darrell, T., and Canny, J. (2023). CLAIR: Evaluating Image Captions with Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore, 6–10 December 2023, Association for Computational Linguistics.10.18653/v1/2023.emnlp-main.841 ↗
30. Learning Transferable Visual Models from Natural Language Supervision
31. Luo, Z., Hu, Z., Xi, Y., Zhang, R., and Ma, J. (2023, January 4–10). I-Tuning: Tuning Frozen Language Models with Image for Lightweight Image Captioning. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096424 ↗
32. Ramos, R., Martins, B., Elliott, D., and Kementchedjhieva, Y. (2023). SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023, IEEE.10.1109/CVPR52729.2023.00278 ↗
33. Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., and Wei, F. (2020). Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. Computer Vision–ECCV 2020, 16th European Conference, Glasgow, UK, 23–28 August 2020, Springer.10.1007/978-3-030-58577-8_8 ↗
34. Wang, Z., Yu, J., Yu, A.W., Dai, Z., Tsvetkov, Y., and Cao, Y. (2022, January 25–29). SimVLM: Simple Visual Language Model Pretraining with Weak Supervision. Proceedings of the International Conference on Learning Representations (ICLR), Virtual.
35. Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. (2022). CoCa: Contrastive Captioners are Image-Text Foundation Models. arXiv.
36. Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O.K., Singhal, S., and Som, S. (2023). Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023, IEEE.10.1109/CVPR52729.2023.01838 ↗