Panoptic Segmentation-Based Attention for Image Captioning
Source abstract
Abstract
Image captioning is the task of generating textual descriptions of images. In order to obtain a better image representation, attention mechanisms have been widely adopted in image captioning. However, in existing models with detection-based attention, the rectangular attention regions are not fine-grained, as they contain irrelevant regions (e.g., background or overlapped regions) around the object, making the model generate inaccurate captions. To address this issue, we propose panoptic segmentation-based attention that performs attention at a mask-level (i.e., the shape of the main part of an instance). Our approach extracts feature vectors from the corresponding segmentation regions, which is more fine-grained than current attention mechanisms. Moreover, in order to process features of different classes independently, we propose a dual-attention module which is generic and can be applied to other frameworks. Experimental results showed that our model could recognize the overlapped objects and understand the scene better. Our approach achieved competitive performance against state-of-the-art methods. We made our code available.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Software Engineering, South China University of Technology, Guangzhou 510006, China
School of Computer Science and Informatics, Cardiff University, Cardiff CF10 3AT, UK
Funding
Pearl River S and T Nova Program of Guangzhou · Fundamental Research Funds for the Central Universities · National Aerospace Science Foundation of China
References (58)
Show source-provided references
1. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
2. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015, January 6–11). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the International Conference on Machine Learning, Lille, France.
3. Jin, J., Fu, K., Cui, R., Sha, F., and Zhang, C. (2015). Aligning where to see and what to tell: image caption with region-based attention and scene factorization. arXiv.
4. Pedersoli, M., Lucas, T., Schmid, C., and Verbeek, J. (2017, January 22–29). Areas of attention for image captioning. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.140 ↗
5. Fang, H., Gupta, S., Iandola, F., Srivastava, R.K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., and Platt, J.C. (2015, January 7–12). From captions to visual concepts and back. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298754 ↗
6. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18–22). Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
7. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
8. Dai, J., He, K., and Sun, J. (2016, January 27–30). Instance-aware semantic segmentation via multi-task network cascades. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.343 ↗
9. Caesar, H., Uijlings, J., and Ferrari, V. (2018, January 18–22). COCO-Stuff: Thing and stuff classes in context. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00132 ↗
10. Kirillov, A., He, K., Girshick, R., Rother, C., and Dollár, P. (2019, January 16–20). Panoptic Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00963 ↗
11. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft COCO: Common objects in context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
12. Yang, Z., Yuan, Y., Wu, Y., Cohen, W.W., and Salakhutdinov, R.R. (2016, January 5–10). Review networks for caption generation. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.
13. You, Q., Jin, H., Wang, Z., Fang, C., and Luo, J. (2016, January 27–30). Image captioning with semantic attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.503 ↗
14. Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., and Chua, T.S. (2017, January 21–26). SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.667 ↗
15. Lu, J., Xiong, C., Parikh, D., and Socher, R. (2017, January 21–26). Knowing when to look: Adaptive attention via a visual sentinel for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.345 ↗
16. Wang, W., Chen, Z., and Hu, H. (February, January 27). Hierarchical attention network for image captioning. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
17. Selective search for object recognition10.1007/s11263-013-0620-5 ↗
18. Jaderberg, M., Simonyan, K., Zisserman, A., and Kavukcuoglu, K. (2015, January 7–12). Spatial transformer networks. Proceedings of the 28th International Conference on Neural Information Processing Systems—NIPS’15, Montreal, QC, Canada.
19. Yang, Z., Zhang, Y.J., ur Rehman, S., and Huang, Y. (2017, January 13–15). Image Captioning with Object Detection and Localization. Proceedings of the International Conference on Image and Graphics, Shanghai, China.10.1007/978-3-319-71589-6_10 ↗
20. Lu, J., Yang, J., Batra, D., and Parikh, D. (2018, January 18–22). Neural Baby Talk. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00754 ↗
21. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster R-CNN: Towards real-time object detection with region proposal networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
22. Latent Dirichlet allocation
23. Aligning where to see and what to tell: Image captioning with region-based attention and scene-specific contexts10.1109/TPAMI.2016.2642953 ↗
24. Visual Genome: Connecting language and vision using crowdsourced dense image annotations10.1007/s11263-016-0981-7 ↗
25. A Fine-Grained Spatial-Temporal Attention Model for Video Captioning10.1109/ACCESS.2018.2879642 ↗
26. Zhang, Z., Wu, Q., Wang, Y., and Chen, F. (2018, January 12–15). Fine-Grained and Semantic-Guided Visual Attention for Image Captioning. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision, Lake Tahoe, NV, USA.10.1109/WACV.2018.00190 ↗
27. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
28. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., and Goel, V. (2017, January 21–26). Self-critical sequence training for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.131 ↗
29. Dai, B., Ye, D., and Lin, D. (2018, January 8–14). Rethinking the form of latent states in image captioning. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01228-1_18 ↗
30. Yang, X., Tang, K., Zhang, H., and Cai, J. (2019, January 16–20). Auto-encoding scene graphs for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01094 ↗
31. Yao, T., Pan, Y., Li, Y., and Mei, T. (, January 8–14). Exploring visual relationship for image captioning. Proceedings of the European Conference on Computer Vision, Munich, Germany, 2018.10.1007/978-3-030-01264-9_42 ↗
32. Liu, X., Li, H., Shao, J., Chen, D., and Wang, X. (2018, January 8–14). Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01267-0_21 ↗
33. Dai, B., and Lin, D. (2017, January 4–9). Contrastive learning for image captioning. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
34. Li, Y., Qi, H., Dai, J., Ji, X., and Wei, Y. (2016). Fully convolutional instance-aware semantic segmentation. arXiv.10.1109/CVPR.2017.472 ↗
35. Liu, S., Jia, J., Fidler, S., and Urtasun, R. (2017, January 22–29). SGN: Sequential grouping networks for instance segmentation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.378 ↗
36. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs10.1109/TPAMI.2017.2699184 ↗
37. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2014). Semantic image segmentation with deep convolutional nets and fully connected CRFs. arXiv.
38. Noh, H., Hong, S., and Han, B. (2015, January 7–13). Learning deconvolution network for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.178 ↗
39. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
40. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015, January 7–12). CIDEr: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
41. Dai, J., He, K., and Sun, J. (2015, January 7–12). Convolutional feature masking for joint object and stuff segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299025 ↗
42. Karpathy, A., and Fei-Fei, L. (2015, January 7–12). Deep visual-semantic alignments for generating image descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298932 ↗
43. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
44. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
45. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
46. Denkowski, M., and Lavie, A. (2014, January 26–27). Meteor universal: Language specific translation evaluation for any target language. Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, MA, USA.10.3115/v1/W14-3348 ↗
47. Lin, C.Y. (2004, January 21–26). ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of the Association for Computational Linguistics Workshop, Barcelona, Spain. Available online: https://www.aclweb.org/anthology/W04-1013.pdf.
48. Re-Caption: Saliency-Enhanced Image Captioning Through Two-Phase Learning10.1109/TIP.2019.2928144 ↗
49. Aneja, J., Deshpande, A., and Schwing, A.G. (2018, January 18–22). Convolutional image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00583 ↗
50. Chen, X., Ma, L., Jiang, W., Yao, J., and Liu, W. (2018, January 18–22). Regularizing RNNs for Caption Generation by Reconstructing the Past With the Present. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00834 ↗
51. Jiang, W., Ma, L., Chen, X., Zhang, H., and Liu, W. (2018, January 2–7). Learning to Guide Decoding for Image Captioning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12283 ↗
52. Dognin, P., Melnyk, I., Mroueh, Y., Ross, J., and Sercu, T. (2019, January 16–20). Adversarial Semantic Alignment for Improved Image Captions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01071 ↗
53. Gu, J., Cai, J., Wang, G., and Chen, T. (2018, January 2–7). Stack-Captioning: Coarse-to-Fine Learning for Image Captioning. Proceedings of theThirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12266 ↗
54. Lu, S., Hu, R., Liu, J., Guo, L., and Zheng, F. (2019). Structure Preserving Convolutional Attention for Image Captioning. Appl. Sci., 9.10.3390/app9142888 ↗
55. Self-Guiding Multimodal LSTM-when we do not have a perfect training dataset for image captioning10.1109/TIP.2019.2917229 ↗
56. Borji, A., and Iranmanesh, S.M. (2019). Empirical Upper-bound in Object Detection and More. arXiv.
57. Divvala, S.K., Hoiem, D., Hays, J.H., Efros, A.A., and Hebert, M. (2009, January 20–25). An empirical study of context in object detection. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPRW.2009.5206532 ↗
58. Heitz, G., and Koller, D. (2008, January 12–18). Learning spatial context: Using stuff to find things. Proceedings of the European Conference on Computer Vision, Marseille, France.10.1007/978-3-540-88682-2_4 ↗