MFEAM: Multi-View Feature Enhanced Attention Model for Image Captioning
Source abstract
Abstract
Image captioning plays a crucial role in aligning visual content with natural language, serving as a key step toward effective cross-modal understanding. Transformer has become the dominant language model in image captioning. Existing Transformer-based models seldom highlight important features from multiple views in the use of self-attention. In this paper, we propose MFEAM, an innovative network that leverages the multi-view feature enhanced attention. To accurately represent the entangled features of vision and text, the teacher model employs the multi-view feature enhanced attention to guide the student model training through knowledge distillation and model averaging from both visual and textual views. To mitigate the impact of excessive feature enhancement, the student model divides the decoding layer into two groups, which separately process instance features and the relationships between instances. Experimental results demonstrate that MFEAM attains competitive performance on the MSCOCO (Microsoft Common Objects in Context) when trained without leveraging external data.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Human Pose and Action Recognition · Advanced Image and Video Retrieval Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China
References (56)
Show source-provided references
1. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
2. Karpathy, A., and Fei-Fei, L. (2015, January 7–12). Deep visual-semantic alignments for generating image descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298932 ↗
3. Aneja, J., Deshpande, A., and Schwing, A.G. (2018, January 18–23). Convolutional image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00583 ↗
4. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18–23). Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
5. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., and Goel, V. (2017, January 21–26). Self-critical sequence training for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.131 ↗
6. Evolutionary recurrent neural network for image captioning10.1016/j.neucom.2020.03.087 ↗
7. Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. (2015, January 7–12). Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.21236/ADA623249 ↗
8. Wang, C., Yang, H., Bartz, C., and Meinel, C. (2016, January 15–19). Image captioning with deep bidirectional LSTMs. Proceedings of the 24th ACM International Conference on Multimedia, Amsterdam, The Netherlands.10.1145/2964284.2964299 ↗
9. Lu, J., Xiong, C., Parikh, D., and Socher, R. (2017, January 21–26). Knowing when to look: Adaptive attention via a visual sentinel for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.345 ↗
10. Image captioning with triple-attention and stack parallel LSTM10.1016/j.neucom.2018.08.069 ↗
11. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, PmLR, Virtual.
12. Geometry Attention Transformer with position-aware LSTMs for image captioning10.1016/j.eswa.2022.117174 ↗
13. Barraco, M., Stefanini, M., Cornia, M., Cascianelli, S., Baraldi, L., and Cucchiara, R. (2022, January 21–25). CaMEL: Mean teacher learning for image captioning. Proceedings of the 2022 26th International Conference on Pattern Recognition (ICPR), Montreal, QC, Canada.10.1109/ICPR56361.2022.9955644 ↗
14. Mobilenet V3-transformer, a lightweight model for image caption
15. Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., and Li, Z. PIXART-sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation. Proceedings of the European Conference on Computer Vision.
16. Moratelli, N., Caffagni, D., Cornia, M., Baraldi, L., and Cucchiara, R. (2024). Revisiting Image Captioning Training Paradigm via Direct CLIP-based Optimization. arXiv.
17. Wang, F., Mei, J., and Yuille, A. Sclip: Rethinking self-attention for dense vision-language inference. Proceedings of the European Conference on Computer Vision.
18. Moratelli, N., Cornia, M., Baraldi, L., and Cucchiara, R. Fluent and Accurate Image Captioning with a Self-Trained Reward Model. Proceedings of the International Conference on Pattern Recognition.
19. Tarvainen, A., and Valpola, H. (2017, January 4–9). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Proceedings of the 31st International Conference on Neural Information Processing System, Long Beach, CA, USA.
20. Gu, Y., Dong, L., Wei, F., and Huang, M. (2024, January 7–11). MiniLLM: Knowledge distillation of large language models. Proceedings of the Twelfth International Conference on Learning Representations, Vienna Austria.
21. Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks
22. Li, Z., Li, X., Fu, X., Zhang, X., Wang, W., Chen, S., and Yang, J. (2024, January 16–22). Promptkd: Unsupervised prompt distillation for vision-language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02513 ↗
23. Improving multimodal datasets with image captioning
24. Mahmoud, A., Elhoushi, M., Abbas, A., Yang, Y., Ardalani, N., Leather, H., and Morcos, A.S. (2024, January 16–22). Sieve: Multimodal dataset pruning using image captioning models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02116 ↗
25. Awadalla, A., Xue, L., Shu, M., Yan, A., Wang, J., Purushwalkam, S., Shen, S., Lee, H., Lo, O., and Park, J.S. (2024). BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions. arXiv.
26. Yu, Q., Sun, Q., Zhang, X., Cui, Y., Zhang, F., Cao, Y., Wang, X., and Liu, J. (2024, January 16–20). Capsfusion: Rethinking image-text data at scale. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01330 ↗
27. Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. Proceedings of the European Conference on Computer Vision.
28. Liao, Y., Zhang, A., Lu, M., Wang, Y., Li, X., and Liu, S. (2022, January 18–24). Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01949 ↗
29. Layer-wise enhanced transformer with multi-modal fusion for image caption10.1007/s00530-022-01036-z ↗
30. Bootstrapping interactive image-text alignment for remote sensing image captioning
31. An efficient automated image caption generation by the encoder decoder model10.1007/s11042-024-18150-x ↗
32. Yao, T., Pan, Y., Li, Y., and Mei, T. (2018, January 8–14). Exploring visual relationship for image captioning. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_42 ↗
33. Attention is all you need
34. Hinton, G. (2015). Distilling the Knowledge in a Neural Network. arXiv.
35. Zhou, Y., Zhang, Y., Hu, Z., and Wang, M. (2021, January 10–17). Semi-autoregressive transformer for image captioning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCVW54120.2021.00350 ↗
36. Sameni, S., Kafle, K., Tan, H., and Jenni, S. (2024, January 16–22). Building Vision-Language Models on Solid Foundations with Masked Distillation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01348 ↗
37. EDIR: An expert method for describing image regions based on knowledge distillation and triple fusion10.1007/s10489-024-06027-3 ↗
38. Zhou, Y., Wang, M., Liu, D., Hu, Z., and Zhang, H. (2020, January 14–19). More grounded image captioning by distilling image-text matching model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00483 ↗
39. Bajpai, D.J., and Hanawal, M.K. (2024). CAPEEN: Image Captioning with Early Exits and Knowledge Distillation. arXiv.10.18653/v1/2024.findings-emnlp.376 ↗
40. ALIGN: A program to superimpose protein coordinates, accounting for insertions and deletions10.1107/S0021889897006729 ↗
41. Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M., Liu, C., and Yuan, L. (2024, January 16–22). Florence-2: Advancing a unified representation for a variety of vision tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00461 ↗
42. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. (2015, January 7–12). Cider: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
43. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft coco: Common objects in context. Proceedings of the Computer Vision—ECCV 2014: 13th European Conference, Zurich, Switzerland. Proceedings, Part V 13.10.1007/978-3-319-10602-1_48 ↗
44. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6–12). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
45. Banerjee, S., and Lavie, A. (2005, January 29). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the acl Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, MI, USA.
46. Lin, C.Y. (2004, January 22). Rouge: A package for automatic evaluation of summaries. Proceedings of the Text Summarization Branches Out, Barcelona, Spain.
47. Anderson, P., Fernando, B., Johnson, M., and Gould, S. (2016, January 11–14). Spice: Semantic propositional image caption evaluation. Proceedings of the Computer Vision—ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part V 14.10.1007/978-3-319-46454-1_24 ↗
48. Sennrich, R., Haddow, B., and Birch, A. (2015). Neural machine translation of rare words with subword units. arXiv.10.18653/v1/P16-1162 ↗
49. Yao, T., Pan, Y., Li, Y., and Mei, T. (November, January 27). Hierarchy parsing for image captioning. Proceedings of the IEEE/CVF INTERNATIONAL Conference on Computer Vision, Seoul, Republic of Korea.
50. Wu, M., Zhang, X., Sun, X., Zhou, Y., Chen, C., Gu, J., Sun, X., and Ji, R. (November, January 27). Difnet: Boosting visual information flow for image captioning. Proceedings of the IEEE/CVF Conference on Computer vision and Pattern Recognition, Seoul, Republic of Korea.
51. Huang, L., Wang, W., Chen, J., and Wei, X.Y. (November, January 27). Attention on attention for image captioning. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
52. Pan, Y., Yao, T., Li, Y., and Mei, T. (2020, January 13–19). X-linear attention networks for image captioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01098 ↗
53. Luo, Y., Ji, J., Sun, X., Cao, L., Wu, Y., Huang, F., Lin, C.W., and Ji, R. (2021, January 2–9). Dual-level collaborative transformer for image captioning. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v35i3.16328 ↗
54. Zhang, X., Sun, X., Luo, Y., Ji, J., Zhou, Y., Wu, Y., Huang, F., and Ji, R. (2021, January 20–25). Rstnet: Captioning with adaptive attention on visual and non-visual words. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01521 ↗
55. BENet: Bi-directional enhanced network for image captioning10.1007/s00530-023-01230-7 ↗
56. Enhance understanding and reasoning ability for image captioning10.1007/s10489-022-03624-y ↗