Middle-Level Attribute-Based Language Retouching for Image Caption Generation
Source abstract
Abstract
Image caption generation is attractive research which focuses on generating natural language sentences to describe the visual content of a given image. It is an interdisciplinary subject combining computer vision (CV) and natural language processing (NLP). The existing image captioning methods are mainly focused on generating the final image caption directly, which may lose significant identification information of objects contained in the raw image. Therefore, we propose a new middle-level attribute-based language retouching (MLALR) method to solve this problem. Our proposed MLALR method uses the middle-level attributes predicted from the object regions to retouch the intermediate image description, which is generated by our language generation model. The advantage of our MLALR method is that it can correct descriptive errors in the intermediate image description and make the final image caption more accurate. Moreover, evaluation using benchmark datasets—MSCOCO, Flickr8K, and Flickr30K—validated the impressive performance of our MLALR method with evaluation metrics—BLEU, METEOR, ROUGE-L, CIDEr, and SPICE.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Video Analysis and Summarization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Mechanical Electronic & Information Engineering, China University of Mining & Technology (Beijing), Beijing 100083, China
G-Cloud Technology Corporation, Cloud Computing Center, Chinese Academy of Sciences, Dongguan 523808, China
Funding
the National Key R&G Program of China · the National Key Research and Development Program of China
References (42)
Show source-provided references
1. Wei, Y., Xia, W., Huang, J., Ni, B., Dong, J., Zhao, Y., and Yan, S. (arXiv, 2014). CNN: Single-label to Multi-label, arXiv.
2. Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., and Khudanpur, S. (2010, January 26–30). Recurrent neural network based language model. Proceedings of the Eleventh Annual Conference of the International Speech Communication Association, Chiba, Japan.10.21437/Interspeech.2010-343 ↗
3. Simonyan, K., and Zisserman, A. (arXiv, 2014). Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv.
4. He, K., Zhang, X., Ren, S., and Sun, J. (arXiv, 2015). Deep Residual Learning for Image Recognition, arXiv.10.1109/CVPR.2016.90 ↗
5. Cho, K., van Merrienboer, B., Gülçehre, Ç., Bougares, F., Schwenk, H., and Bengio, Y. (arXiv, 2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, arXiv.10.3115/v1/D14-1179 ↗
6. Bidirectional recurrent neural networks10.1109/78.650093 ↗
7. Hochreiter, S., and Schmidhuber, J. (1997). Long Short-term Memory. Neural Computation, MIT Press.10.1162/neco.1997.9.8.1735 ↗
8. Ren, S., He, K., Girshick, R.B., and Sun, J. (arXiv, 2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal, arXiv.
9. Papineni, K., Roukos, S., and Ward, T. (2002, January 7–12). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
10. Denkowski, M., and Lavie, A. (2014, January 26–27). Meteor Universal: Language Specific Translation Evaluation for Any Target Language. Proceedings of the Ninth Workshop on Statistical Machine Translation, Association for Computational Linguistics, Baltimore, MD, USA.10.3115/v1/W14-3348 ↗
11. Lin, C.Y. (2004, January 25–26). ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of the ACL-04 Workshop: Text Summarization Branches Out, Barcelona, Spain.
12. Vedantam, R., Zitnick, C.L., and Parikh, D. (arXiv, 2014). CIDEr: Consensus-based Image Description Evaluation, arXiv.10.1109/CVPR.2015.7299087 ↗
13. Anderson, P., Fernando, B., Johnson, M., and Gould, S. (2016). SPICE: Semantic Propositional Image Caption Evaluation. ECCV, Springer.10.1007/978-3-319-46454-1_24 ↗
14. Mao, J., Xu, W., Yang, Y., Wang, J., and Yuille, A.L. (arXiv, 2014). Explain Images with Multimodal Recurrent Neural Networks, arXiv.
15. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
16. Deep Visual-Semantic Alignments for Generating Image Descriptions10.1109/TPAMI.2016.2598339 ↗
17. Long-Term Recurrent Convolutional Networks for Visual Recognition and Description10.1109/TPAMI.2016.2599174 ↗
18. You, Q., Jin, H., Wang, Z., Fang, C., and Luo, J. (2016, January 27–30). Image Captioning with Semantic Attention. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.503 ↗
19. Aligning Where to See and What to Tell: Image Captioning with Region-Based Attention and Scene-Specific Contexts10.1109/TPAMI.2016.2642953 ↗
20. A region-based image caption generator with refined descriptions10.1016/j.neucom.2017.07.014 ↗
21. Kiros, R., Salakhutdinov, R., and Zemel, R. (2014, January 21–26). Multimodal neural language models. Proceedings of the International Conference on Machine Learning, Beijing, China.
22. Chen, M., Ding, G., Zhao, S., Chen, H., Liu, Q., and Han, J. (2017, January 4–9). Reference Based LSTM for Image Captioning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11198 ↗
23. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., and Bengio, Y. (arXiv, 2015). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention, arXiv.
24. Lu, J., Xiong, C., Parikh, D., and Socher, R. (2017, January 21–26). Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.345 ↗
25. GLA: Global-Local Attention for Image Description10.1109/TMM.2017.2751140 ↗
26. Pedersoli, M., Lucas, T., Schmid, C., and Verbeek, J. (arXiv, 2016). Areas of Attention for Image Captioning, arXiv.10.1109/ICCV.2017.140 ↗
27. Liu, C., Sun, F., Wang, C., Wang, F., and Yuille, A.L. (arXiv, 2017). MAT: A Multimodal Attentive Translator for Image Captioning, arXiv.10.24963/ijcai.2017/563 ↗
28. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (arXiv, 2017). Bottom-Up and Top-Down Attention for Image Captioning and VQA, arXiv.10.1109/CVPR.2018.00636 ↗
29. Wu, Q., Shen, C., Liu, L., Dick, A., and van den Hengel, A. (2016, January 27–30). What Value Do Explicit High Level Concepts Have in Vision to Language Problems?. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition CVPR, Las Vegas, NV, USA.10.1109/CVPR.2016.29 ↗
30. Yao, T., Pan, Y., Li, Y., Qiu, Z., and Mei, T. (2017, January 22–29). Boosting Image Captioning with Attributes. Proceedings of the IEEE International Conference on Computer Vision ICCV, Venice, Italy.10.1109/ICCV.2017.524 ↗
31. Hu, C., Miao, J., Su, Z., Shi, X., Chen, Q., and Luo, X. (2017, January 1–4). Precision-Enhanced Image Attribute Prediction Model. Proceedings of the 2017 IEEE Trustcom/BigDataSE/ICESS, Sydney, Australia.10.1109/Trustcom/BigDataSE/ICESS.2017.324 ↗
32. Wang, L., Chu, X., Zhang, W., Wei, Y., Sun, W., and Wu, C. (2018). Social Image Captioning: Exploring Visual Attention and User Attention. Sensors, 18.10.3390/s18020646 ↗
33. Pan, Y., Yao, T., Li, H., and Mei, T. (arXiv, 2016). Video Captioning with Transferred Semantic Attributes, arXiv.10.1109/CVPR.2017.111 ↗
34. Video Captioning with Attention-Based LSTM and Semantic Consistency10.1109/TMM.2017.2729019 ↗
35. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
36. Kumar, N., Berg, A.C., Belhumeur, P.N., and Nayar, S.K. (October, January 29). Attribute and simile classifiers for face verification. Proceedings of the 12th International Conference on Computer Vision ICCV, Kyoto, Japan.
37. Russakovsky, O., and Li, F.F. (2012, January 10–11). Attribute Learning in Large-Scale Datasets. Proceedings of the Trends and Topics in Computer Vision, Crete, Greece.10.1007/978-3-642-35749-7_1 ↗
38. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., and Goel, V. (arXiv, 2016). Self-critical Sequence Training for Image Captioning, arXiv.10.1109/CVPR.2017.131 ↗
39. Lin, T., Maire, M., Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (arXiv, 2014). Microsoft COCO: Common Objects in Context, arXiv.10.1007/978-3-319-10602-1_48 ↗
40. Framing Image Description As a Ranking Task: Data, Models and Evaluation Metrics10.1613/jair.3994 ↗
41. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions10.1162/tacl_a_00166 ↗
42. Fine-grained attention for image caption generation10.1007/s11042-017-4593-1 ↗