A Systematic Literature Review on Image Captioning
Source abstract
Abstract
Natural language problems have already been investigated for around five years. Recent progress in artificial intelligence (AI) has greatly improved the performance of models. However, the results are still not sufficiently satisfying. Machines cannot imitate human brains and the way they communicate, so it remains an ongoing task. Due to the increasing amount of information on this topic, it is very difficult to keep on track with the newest researches and results achieved in the image captioning field. In this study a comprehensive Systematic Literature Review (SLR) provides a brief overview of improvements in image captioning over the last four years. The main focus of the paper is to explain the most common techniques and the biggest challenges in image captioning and to summarize the results from the newest papers. Inconsistent comparison of results achieved in image captioning was noticed during this study and hence the awareness of incomplete data collection is raised in this paper. Therefore, it is very important to compare results of a newly created model produced with the newest information and not only with the state of the art methods. This SLR is a source of such information for researchers in order for them to be precisely correct on result comparison before publishing new achievements in the image caption generation field.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Technologies, Vilnius Gediminas Technical University, Saulėtekio al. 11, LT-10223 Vilnius, Lithuania
References (96)
Show source-provided references
1. Kiros, R., Salakhutdinov, R., and Zemel, R. (2014, January 21–26). Multimodal neural language models. Proceedings of the International Conference on Machine Learning, Beijing, China.
2. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
3. Image captioning based on a hierarchical attention mechanism and policy gradient optimization
4. Karpathy, A., and Li, F.-F. (2015, January 7–12). Deep visual-semantic alignments for generating image descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298932 ↗
5. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., and Bengio, Y. (2015, January 6–11). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the 32nd International Conference on Machine Learning (ICML’15), Lille, France.
6. A comprehensive survey of deep learning for image captioning10.1145/3295748 ↗
7. A survey on automatic image caption generation10.1016/j.neucom.2018.05.080 ↗
8. Shabir, S., and Arafat, S.Y. (2018, January 9–10). An image conveys a message: A brief survey on image description generation. Proceedings of the 2018 1st International Conference on Power, Energy and Smart Grid (ICPESG), Azad Kashmir, Pakistan.10.1109/ICPESG.2018.8384519 ↗
9. A survey of neural network algorithms used for image annotation
10. Fan, C., and Crandall, D.J. (2016, January 11–14). DeepDiary: Automatic caption generation for lifelogging image streams. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46604-0_33 ↗
11. Wang, C., Yang, H., Bartz, C., and Meinel, C. (2016, January 15–19). Image captioning with deep bidirectional LSTMs. Proceedings of the 24th ACM International Conference on Multimedia, Amsterdam, The Netherlands.10.1145/2964284.2964299 ↗
12. Mathews, A.P., Xie, L., and He, X. (2016, January 12–17). Senticap: Generating image descriptions with sentiments. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10475 ↗
13. Sugano, Y., and Bulling, A. (2016). Seeing with humans: Gaze-assisted neural image captioning. arXiv.
14. Venugopalan, S., Anne Hendricks, L., Rohrbach, M., Mooney, R., Darrell, T., and Saenko, K. (2017, January 21–26). Captioning images with diverse objects. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.130 ↗
15. Image caption generation with part of speech guidance10.1016/j.patrec.2017.10.018 ↗
16. Vedantam, R., Bengio, S., Murphy, K., Parikh, D., and Chechik, G. (2017, January 21–26). Context-aware captions from context-agnostic supervision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.120 ↗
17. Shetty, R., Rohrbach, M., Anne Hendricks, L., Fritz, M., and Schiele, B. (2017, January 22–29). Speaking the same language: Matching machine to human captions by adversarial training. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.445 ↗
18. Xin, M., Zhang, H., Yuan, D., and Sun, M. (2017, January 10–14). Learning discriminative action and context representations for action recognition in still images. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, China.10.1109/ICME.2017.8019399 ↗
19. Yuan, A., Li, X., and Lu, X. (2017, January 11–14). FFGS: Feature fusion with gating structure for image caption generation. Proceedings of the Computer Vision, Tianjin, China.10.1007/978-981-10-7299-4_53 ↗
20. Yang, L., Tang, K., Yang, J., and Li, L.J. (2017, January 21–26). Dense captioning with joint inference and visual context. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.214 ↗
21. Data-driven image captioning via salient region discovery10.1049/iet-cvi.2016.0286 ↗
22. Shah, P., Bakrola, V., and Pati, S. (2017, January 17–18). Image captioning using deep neural architectures. Proceedings of the 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), Coimbatore, India.10.1109/ICIIECS.2017.8276124 ↗
23. Huang, Q., Smolensky, P., He, X., Deng, L., and Wu, D. (2018, January 1–6). Tensor product generation networks for deep NLP modeling. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), New Orleans, LA, USA.10.18653/v1/N18-1114 ↗
24. MAT: A multimodal translator for image captioning
25. Ren, Z., Wang, X., Zhang, N., Lv, X., and Li, L.-J. (2017, January 21–26). Deep reinforcement learning-based image captioning with embedding reward. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.128 ↗
26. Pedersoli, M., Lucas, T., Schmid, C., and Verbeek, J. (2017, January 22–29). Areas of attention for image captioning. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.140 ↗
27. Aligning where to see and what to tell: Image captioning with region-based attention and scene-specific contexts10.1109/TPAMI.2016.2642953 ↗
28. Yao, T., Pan, Y., Li, Y., Qiu, Z., and Mei, T. (2017, January 22–29). Boosting image captioning with attributes. Proceedings of the IEEE Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.524 ↗
29. Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge10.1109/TPAMI.2016.2587640 ↗
30. Xu, K., Wang, H., and Tang, P. (2017, January 10–14). Image captioning with deep LSTM based on sequential residual. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, China.10.1109/ICME.2017.8019408 ↗
31. Dai, B., Fidler, S., Urtasun, R., and Lin, D. (2017, January 22–29). Towards diverse and natural image descriptions via a conditional GAN. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.323 ↗
32. Dai, B., and Lin, D. (2017). Contrastive learning for image captioning. Advances in Neural Information Processing Systems, MIT Press.
33. Liu, C., Sun, F., Wang, C., Wang, F., and Yuille, A. (2017, January 19–25). MAT: A multimodal attentive translator for image captioning. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, Melbourne, Australia.10.24963/ijcai.2017/563 ↗
34. Lu, J., Xiong, C., Parikh, D., and Socher, R. (2017, January 21–26). Knowing when to look: Adaptive attention via a visual sentinel for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.345 ↗
35. Nguyen, D.-K., and Okatani, T. (2018). Multi-task learning of hierarchical vision-language representation. arXiv.10.1109/CVPR.2019.01074 ↗
36. Zhang, J., Shih, K., Tao, A., Catanzaro, B., and Elgammal, A. (2018). An interpretable model for scene graph generation. arXiv.
37. Yan, S., Wu, F., Smith, J.S., Lu, W., and Zhang, B. (2018, January 20–24). Image captioning using adversarial networks and reinforcement learning. Proceedings of the 24th International Conference on Pattern Recognition (ICPR), Beijing, China.10.1109/ICPR.2018.8545049 ↗
38. Liu, D., Fu, J., Qu, Q., and Lv, J. (2018). BFGAN: Backward and forward generative adversarial networks for lexically constrained sentence generation. arXiv.10.1109/TASLP.2019.2943018 ↗
39. Wu, Y., Zhu, L., Jiang, L., and Yang, Y. (2018, January 22–26). Decoupled novel object captioner. Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Korea.10.1145/3240508.3240640 ↗
40. Wang, W., Ding, Y., and Tian, C. (2018, January 15–20). A novel semantic attribute-based feature for image caption generation. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461507 ↗
41. Fine-grained attention for image caption generation10.1007/s11042-017-4593-1 ↗
42. Image captioning and visual question answering based on attributes and external knowledge10.1109/TPAMI.2017.2708709 ↗
43. Baig, M.M.A., Shah, M.I., Wajahat, M.A., Zafar, N., and Arif, O. (2018, January 10–13). Image caption generator with novel object injection. Proceedings of the 2018 Digital Image Computing: Techniques and Applications (DICTA), Canberra, Australia.10.1109/DICTA.2018.8615810 ↗
44. Chen, F., Ji, R., Sun, X., Wu, Y., and Su, J. (2018, January 18–22). GroupCap: Group-based image captioning with structured relevance and diversity constraints. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00146 ↗
45. A region-based image caption generator with refined descriptions10.1016/j.neucom.2017.07.014 ↗
46. Cornia, M., Baraldi, L., and Cucchiara, R. (2018). Show, control and tell: A framework for generating controllable and grounded captions. arXiv.10.1109/CVPR.2019.00850 ↗
47. Bi-directional spatial-semantic attention networks for image-text matching10.1109/TIP.2018.2882225 ↗
48. Chen, T., Zhang, Z., You, Q., Fang, C., Wang, Z., Jin, H., and Luo, J. (2018, January 8–14). “Factual” or “emotional”: Stylized image captioning with adaptive learning and attention. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01249-6_32 ↗
49. You, Q., Jin, H., and Luo, J. (2018). Image captioning at will: A versatile scheme for effectively injecting sentiments into image descriptions. arXiv.
50. Mathews, A., Xie, L., and He, X. (2018, January 18–22). SemStyle: Learning to generate stylised image captions using unaligned text. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00896 ↗
51. Wang, Q., and Chan, A.B. (2018). CNN+CNN: Convolutional decoders for image captioning. arXiv.
52. Aneja, J., Deshpande, A., and Schwing, A. (2018, January 18–22). Convolutional image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00583 ↗
53. Paying more attention to saliency: Image captioning with saliency and context attention
54. Shi, H., and Li, P. (2018, January 17–19). Image captioning based on deep reinforcement learning. Proceedings of the 10th International Conference on Internet Multimedia Computing and Service, Nanjing, China.10.1145/3240876.3240900 ↗
55. Fan, Y., Xu, J., Sun, Y., and He, B. (2018, January 5–7). Long-term recurrent merge network model for image captioning. Proceedings of the 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI), Volos, Greece.10.1109/ICTAI.2018.00047 ↗
56. Shen, K., Kar, A., and Fidler, S. (2018). Lifelong learning for image captioning by asking natural language questions. arXiv.
57. Topic-oriented image captioning based on order-embedding10.1109/TIP.2018.2889922 ↗
58. Kim, B., Lee, Y.H., Jung, H., and Cho, C. (2018). Distinctive-attribute extraction for image captioning. Eur. Conf. Comput. Vis., 133–144.10.1007/978-3-030-11018-5_12 ↗
59. Delbrouck, J.-B., and Dupont, S. (2018). Bringing back simplicity and lightliness into neural image captioning. arXiv.
60. Sow, D., Qin, Z., Niasse, M., and Wan, T. (2018). A sequential guiding network with attention for image captioning. arXiv.10.1109/ICASSP.2019.8682505 ↗
61. Image captioning with triple-attention and stack parallel LSTM10.1016/j.neucom.2018.08.069 ↗
62. Looking deeper and transferring attention for image captioning10.1007/s11042-018-6228-6 ↗
63. Fang, F., Wang, H., and Tang, P. (2018, January 7–10). Image captioning with word level attention. Proceedings of the 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.10.1109/ICIP.2018.8451558 ↗
64. Zhu, X., Li, L., Liu, J., Peng, H., and Niu, X. (2018). Captioning transformer with stacked attention modules. Appl. Sci., 8.10.3390/app8050739 ↗
65. Wang, F., Gong, X., and Huang, L. (2018, January 20–24). Time-dependent pre-attention model for image captioning. Proceedings of the 24th International Conference on Pattern Recognition (ICPR), Beijing, China.10.1109/ICPR.2018.8545355 ↗
66. Ren, L., and Hua, K. (2018, January 10–12). Improved image description via embedded object structure graph and semantic feature matching. Proceedings of the IEEE International Symposium on Multimedia (ISM), Taichung, Taiwan.10.1109/ISM.2018.00021 ↗
67. Jiang, W., Ma, L., Jiang, Y.G., and Liu, W. (2018, January 8–14). Recurrent fusion network for image captioning. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01216-8_31 ↗
68. Multitask learning for cross-domain image captioning10.1109/TMM.2018.2869276 ↗
69. Chen, C., Mu, S., Xiao, W., Ye, Z., Wu, L., and Ju, Q. (2018). Improving image captioning with conditional generative adversarial nets. arXiv.10.1609/aaai.v33i01.33018142 ↗
70. Du, J., Qin, Y., Lu, H., and Zhang, Y. (2018). Attend more times for image captioning. arXiv.
71. 3G structure for image caption generation10.1016/j.neucom.2018.10.059 ↗
72. Kim, D.-J., Choi, J., Oh, T.-H., and Kweon, I.S. (2019, January 15–21). Dense relational captioning: Triple-stream networks for relationship-based captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00643 ↗
73. Mishra, A., and Liwicki, M. (2019). Using deep object features for image descriptions. arXiv.
74. Scene graph captioner: Image captioning based on structural visual representation10.1016/j.jvcir.2018.12.027 ↗
75. Phrase-based image caption generator with hierarchical LSTM network10.1016/j.neucom.2018.12.026 ↗
76. Tan, J.H., Chan, C.S., and Chuah, J.H. (2019). COMIC: Towards a compact image captioning model with attention. IEEE Trans. Multimed.10.1109/TMM.2019.2904878 ↗
77. Image captioning with text-based visual attention10.1007/s11063-018-9807-7 ↗
78. Li, J., Ebrahimpour, M.K., Moghtaderi, A., and Yu, Y.-Y. (2019). Image captioning with weakly-supervised attention penalty. arXiv.
79. VD-SAN: Visual-densely semantic attention network for image caption generation10.1016/j.neucom.2018.02.106 ↗
80. A multimodal fusion approach for image captioning10.1016/j.neucom.2018.11.004 ↗
81. Wang, W., and Hu, H. (2019). Image captioning using region-based attention joint with time-varying attention. Neural Process. Lett., 1–13.10.1007/s11063-019-10005-z ↗
82. Zhou, Y., Sun, Y., and Honavar, V. (2019, January 7–11). Improving image captioning by leveraging knowledge graphs. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa Village, HI, USA.10.1109/WACV.2019.00036 ↗
83. Ren, L., Qi, G., and Hua, K. (2019, January 7–11). Improving diversity of image captioning through variational autoencoders and adversarial learning. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa Village, HI, USA.10.1109/WACV.2019.00034 ↗
84. More is better: Precise and detailed image captioning using online positive recall and missing concepts mining10.1109/TIP.2018.2855415 ↗
85. Li, X., and Jiang, S. (2019). Know more say less: Image captioning based on scene graphs. IEEE Trans. Multimed.10.1109/TMM.2019.2896516 ↗
86. Gao, L., Li, X., Song, J., and Shen, H.T. (2019). Hierarchical LSTMs with adaptive attention for visual captioning. IEEE Trans. Pattern Anal. Mach. Intell.10.1109/TPAMI.2019.2894139 ↗
87. Zha, Z.J., Liu, D., Zhang, H., Zhang, Y., and Wu, F. (2019). Context-aware visual policy network for fine-grained image captioning. IEEE Trans. Pattern Anal. Mach. Intell.
88. KOUSTUBH (2019, May 24). ResNet, AlexNet, VGGNet, Inception: Understanding Various Architectures of Convolutional Networks. Available online: https://cv-tricks.com/cnn/understand-resnet-alexnet-vgg-inception/.
89. He, S., Tavakoli, H.R., Borji, A., and Pugeault, N. (2019). A synchronized multi-modal attention-caption dataset and analysis. arXiv.10.1109/ICCV.2019.00862 ↗
90. Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., and Lazebnik, S. (2015, January 7–13). Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.303 ↗
91. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft COCO: Common objects in context. Proceedings of the Computer Vision—ECCV 2014, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
92. Large scale retrieval and generation of image descriptions10.1007/s11263-015-0840-y ↗
93. Agrawal, H., Desai, K., Chen, X., Jain, R., Batra, D., Parikh, D., Lee, S., and Anderson, P. (2018). Nocaps: Novel object captioning at scale. arXiv.10.1109/ICCV.2019.00904 ↗
94. A context-driven extractive framework for generating realistic image descriptions10.1109/TIP.2016.2628585 ↗
95. Learning from collective intelligence: Feature learning using social images and tags
96. Sharma, S., Suhubdy, D., Michalski, V., Kahou, S.E., and Bengio, Y. (2018). ChatPainter: Improving text to image generation using dialogue. arXiv.