NuCap: A Numerically Aware Captioning Framework for Improved Numerical Reasoning
Source abstract
Abstract
Despite advances in image captioning, existing models struggle to generate captions that include accurate numerical information, especially the number of objects. One reason for this issue is that the dataset used for training has a limited number of samples with numerical information about the image. To address this issue, we propose a new framework, the Numerically Aware Captioning (NuCap) model, to enhance numerical reasoning in caption generation. We extract dual features by combining a region-attended object encoder for finer-grained object features and a spatially attended grid encoder for encoding spatially distributed global features. We also propose a number-focused cross-entropy loss component to increase sensitivity to numerical tokens, and introduce CountCOCO, a dataset for structured understanding of numerical information. Experiments show that our method achieves statistically significant counting performance improvements over state-of-the-art image captioning models while maintaining similar captioning performance. Despite the significant improvement in numerical reasoning power, our proposed approach has significantly fewer parameters and lower inference latency than large-scale vision language models, demonstrating both computational efficiency and stability. NuCap is an image captioning model that can represent specific numerical information in a given image, making it more suitable for applications that require precise object enumeration, such as automated surveillance, store monitoring, and scientific documentation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Natural Language Processing Techniques · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence, Hanyang University, Seoul 04763, Republic of Korea
Department of Computer Science and Engineering, Hanyang University, Seoul 04763, Republic of Korea
Funding
National Research Foundation of Korea (NRF) · Institute of Information and communications Technology Planning and evaluation (IITP)
References (37)
Show source-provided references
1. From show to tell: A survey on deep learning-based image captioning10.1109/TPAMI.2022.3148210 ↗
2. O’shea, K., and Nash, R. (2015). An introduction to convolutional neural networks. arXiv.
3. Attention is all you need
4. Yang, X., Liu, Y., and Wang, X. (2022, January 10–14). Reformer: The relational transformer for image captioning. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3548409 ↗
5. Cornia, M., Stefanini, M., Baraldi, L., and Cucchiara, R. (2020, January 13–19). Meshed-memory transformer for image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01059 ↗
6. Multimodal transformer with multi-view visual representation for image captioning10.1109/TCSVT.2019.2947482 ↗
7. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18–22). Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
8. Huang, L., Wang, W., Chen, J., and Wei, X.Y. (November, January 27). Attention on attention for image captioning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
9. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014). Microsoft coco: Common objects in context. Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, 6–12 September 2014, Springer.10.1007/978-3-319-10602-1_48 ↗
10. Visual genome: Connecting language and vision using crowdsourced dense image annotations10.1007/s11263-016-0981-7 ↗
11. Hu, J.C., Cavicchioli, R., and Capotondi, A. (2023). Exploiting multiple sequence lengths in fast end to end training for image captioning. Proceedings of the 2023 IEEE International Conference on Big Data (BigData), Sorrento, Italy, 15–18 December 2023, IEEE.10.1109/BigData59044.2023.10386812 ↗
12. Zeng, Z., Xie, Y., Zhang, H., Chen, C., Chen, B., and Wang, Z. (2024, January 17–21). Meacap: Memory-augmented zero-shot image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01337 ↗
13. Towards retrieval-augmented architectures for image captioning10.1145/3663667 ↗
14. Liu, G., Li, Y., Fei, Z., Fu, H., Luo, X., and Guo, Y. (2023). Prefix-diffusion: A lightweight diffusion model for diverse image captioning. arXiv.
15. Deep-learning-based counting methods, datasets, and applications in agriculture: A review10.1007/s11119-023-10034-8 ↗
16. Jiang, R., Liu, L., and Chen, C. (November, January 29). Clip-count: Towards text-guided zero-shot object counting. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
17. Countgd: Multi-modal open-world counting
18. Huang, Z., Dai, M., Zhang, Y., Zhang, J., and Shan, H. (2024, January 17–21). Point Segment and Count: A Generalized Framework for Object Counting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01615 ↗
19. Wang, Z., Li, Y., Wan, J., and Vasconcelos, N. (2025). Diffusion-based data augmentation for object counting problems. Proceedings of the ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hyderabad, India, 6–11 April 2025, IEEE.10.1109/ICASSP49660.2025.10888449 ↗
20. Li, J., Li, D., Xiong, C., and Hoi, S. (2022, January 17–23). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. Proceedings of the International Conference on Machine Learning. PMLR, Baltimore, MD, USA.
21. Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., and Zhou, J. (2024, January 7–11). mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. Proceedings of the Thirteenth International Conference on Learning Representations, Vienna, Austria.
22. What matters when building vision-language models?
23. Evaluating numerical reasoning in text-to-image models
24. Paiss, R., Ephrat, A., Tov, O., Zada, S., Mosseri, I., Irani, M., and Dekel, T. (2023, January 2–3). Teaching clip to count to ten. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00294 ↗
25. Qharabagh, M.F., Ghofrani, M., and Fountoulakis, K. (2024). LVLM-COUNT: Enhancing the Counting Ability of Large Vision-Language Models. arXiv.
26. Nguyen, V.Q., Suganuma, M., and Okatani, T. (2022). Grit: Faster and better image captioning transformer using dual visual features. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23–27 October 2022, Springer.10.1007/978-3-031-20059-5_10 ↗
27. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11–17). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.00986 ↗
28. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2020). Deformable detr: Deformable transformers for end-to-end object detection. arXiv.
29. Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8–14). Cbam: Convolutional block attention module. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_1 ↗
30. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., and Goel, V. (2017, January 21–26). Self-critical sequence training for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.131 ↗
31. Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., and Ge, W. (2024). Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv.
32. Karpathy, A., and Fei-Fei, L. (2017, January 21–26). Deep visual-semantic alignments for generating image descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/TPAMI.2016.2598339 ↗
33. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6–12). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
34. Banerjee, S., and Lavie, A. (2005, January 29). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA.
35. Lin, C.Y. (2004, January 25–26). Rouge: A package for automatic evaluation of summaries. Proceedings of the Text Summarization Branches Out, Barcelona, Spain.
36. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015, January 7–12). Cider: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
37. Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L. (2022). Git: A generative image-to-text transformer for vision and language. arXiv.