Captioning Transformer with Stacked Attention Modules
Source abstract
Abstract
Image captioning is a challenging task. Meanwhile, it is important for the machine to understand the meaning of an image better. In recent years, the image captioning usually use the long-short-term-memory (LSTM) as the decoder to generate the sentence, and these models show excellent performance. Although the LSTM can memorize dependencies, the LSTM structure has complicated and inherently sequential across time problems. To address these issues, recent works have shown benefits of the Transformer for machine translation. Inspired by their success, we develop a Captioning Transformer (CT) model with stacked attention modules. We attempt to introduce the Transformer to the image captioning task. The CT model contains only attention modules without the dependencies of the time. It not only can memorize dependencies between the sequence but also can be trained in parallel. Moreover, we propose the multi-level supervision to make the Transformer achieve better performance. Extensive experiments are carried out on the challenging MSCOCO dataset and the proposed Captioning Transformer achieves competitive performance compared with some state-of-the-art methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Information Security Center, State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China
National Engineering Laboratory for Disaster Backup and Recovery, Beijing University of Posts and Telecommunications, Beijing 100876, China
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
Funding
the National Key Research and Development Program of China · the National Natural Science Foundation of China
References (20)
Show source-provided references
1. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge10.1109/TPAMI.2016.2587640 ↗
2. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention Is All You Need. Proceedings of the NIPS 2017: The Thirty-first Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA.
4. Karpathy, A., and Feifei, L. (2015). Deep visual-semantic alignments for generating image descriptions. Comput. Vis. Pattern Recognit., 3128–3137.10.1109/CVPR.2015.7298932 ↗
5. Jia, X., Gavves, E., Fernando, B., and Tuytelaars, T. (2015). Guiding the Long-Short Term Memory Model for Image Caption Generation. Int. Conf. Learn. Represent., 2407–2415.10.1109/ICCV.2015.277 ↗
6. Yang, Z., Yuan, Y., Wu, Y., Salakhutdinov, R., and Cohen, W.W. (arXiv, 2016). Encode, Review, and Decode: Reviewer Module for Caption Generation, arXiv.
7. Wu, Q., Shen, C., Liu, L., Dick, A.R., and Den Hengel, A.V. (2016). What Value Do Explicit High Level Concepts Have in Vision to Language Problems. Comput. Vis. Pattern Recognit., 203–212.10.1109/CVPR.2016.29 ↗
8. Yao, T., Pan, Y., Li, Y., Qiu, Z., and Mei, T. (2017, January 22–29). Boosting Image Captioning with Attributes. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.524 ↗
9. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (arXiv, 2017). Bottom-up and top-down attention for image captioning and vqa, arXiv.10.1109/CVPR.2018.00636 ↗
10. Xie, S., Girshick, R.B., Dollar, P., Tu, Z., and He, K. (2016). Aggregated Residual Transformations for Deep Neural Networks. Comput. Vis. Pattern Recognit., 1492–1500.10.1109/CVPR.2017.634 ↗
11. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
12. Lin, T., Maire, M., Belongie, S.J., Hays, J., Perona, P., Ramanan, D., Dollar, P., and Zitnick, C.L. (2014, January 6–12). Microsoft COCO: Common Objects in Context. Proceedings of the 13th European Conference, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
13. Vedantam, R., Zitnick, C.L., and Parikh, D. (2015). CIDEr: Consensus-based image description evaluation. Comput. Sci., 4566–4575.10.1109/CVPR.2015.7299087 ↗
14. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, Pennsylvania.10.3115/1073083.1073135 ↗
15. Meteor Universal: Language Specific Translation Evaluation for Any Target Language
16. Kingma, D., and Ba, J. (arXiv, 2014). Adam: A method for stochastic optimization, arXiv.
17. Pu, Y., Gan, Z., Henao, R., Yuan, X., Li, C., Stevens, A., and Carin, L. (2016, January 5–10). Variational Autoencoder for Deep Learning of Images, Labels and Captions. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain.
18. Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, L., and Wang, G. (2015). Recent Advances in Convolutional Neural Networks. Comput. Sci.
19. Liu, S., Zhu, Z., Ye, N., Guadarrama, S., and Murphy, K. (arXiv, 2016). Improved Image Captioning via Policy Gradient optimization of SPIDEr, arXiv.10.1109/ICCV.2017.100 ↗
20. Lu, J., Xiong, C., Parikh, D., and Socher, R. (arXiv, 2016). Knowing when to look: Adaptive attention via A visual sentinel for image captioning, arXiv.10.1109/CVPR.2017.345 ↗