A Review of Transformer-Based Approaches for Image Captioning
Source abstract
Abstract
Visual understanding is a research area that bridges the gap between computer vision and natural language processing. Image captioning is a visual understanding task in which natural language descriptions of images are automatically generated using vision-language models. The transformer architecture was initially developed in the context of natural language processing and quickly found application in the domain of computer vision. Its recent application to the task of image captioning has resulted in markedly improved performance. In this paper, we briefly look at the transformer architecture and its genesis in attention mechanisms. We more extensively review a number of transformer-based image captioning models, including those employing vision-language pre-training, which has resulted in several state-of-the-art models. We give a brief presentation of the commonly used datasets for image captioning and also carry out an analysis and comparison of the transformer-based captioning models. We conclude by giving some insights into challenges as well as future directions for research in this area.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques · Domain Adaptation and Few-Shot Learning
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical and Information Engineering, University of Nairobi, Nairobi P.O. Box 30197-00100, Kenya
Funding
African Development Bank (AfDB) · National Research Fund (NRF) Kenya-South Africa
References (100+)
Show source-provided references
1. Daniilidis, K., Maragos, P., and Paragios, N. (2010, January 5–11). Every Picture Tells a Story: Generating Sentences from Images. Proceedings of the Computer Vision—ECCV 2010, Crete, Greece.10.1007/978-3-642-15567-3 ↗
2. Kulkarni, G., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., and Berg, T.L. (2011, January 20–25). Baby Talk: Understanding and Generating Simple Image Descriptions. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.10.1109/CVPR.2011.5995466 ↗
3. Mitchell, M., Han, X., Dodge, J., Mensch, A., Goyal, A., Berg, A., Yamaguchi, K., Berg, T., Stratos, K., and Daumé, H. (2012, January 23–27). Midge: Generating Image Descriptions from Computer Vision Detections. Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France.
4. Kuznetsova, P., Ordonez, V., Berg, A., Berg, T., and Choi, Y. (2012, January 8–14). Collective Generation of Natural Image Descriptions. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jeju Island, Republic of Korea.
5. TreeTalk: Composition and Compression of Trees for Image Descriptions10.1162/tacl_a_00188 ↗
6. Gradient-Based Learning Applied to Document Recognition10.1109/5.726791 ↗
7. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., and Bernstein, M. (2014). ImageNet Large Scale Visual Recognition Challenge. arXiv.10.1007/s11263-015-0816-y ↗
8. Krizhevsky, A., Sutskever, I., and Hinton, G. (2012, January 3–6). Imagenet Classification with Deep Convolutional Neural Networks. Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
9. Zeiler, M.D., and Fergus, R. (2013). Visualizing and Understanding Convolutional Networks. arXiv.10.1007/978-3-319-10590-1_53 ↗
10. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
11. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2014). Going Deeper with Convolutions. arXiv.10.1109/CVPR.2015.7298594 ↗
12. He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.10.1109/CVPR.2016.90 ↗
13. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 24–27). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the 2014 IEEE Conference on Computer Visionand Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
14. Girshick, R. (2015). Fast R-CNN. arXiv.10.1109/ICCV.2015.169 ↗
15. Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., and Garnett, R. (2015). Advances in Neural Information Processing Systems 28, Curran Associates, Inc.
16. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask R-CNN. arXiv.10.1109/ICCV.2017.322 ↗
17. Kiros, R., Salakhutdinov, R., and Zemel, R. (2014, January 21–26). Multimodal Neural Language Models. Proceedings of the 31st International Conference on Machine Learning—Volume 32, Beijing, China.
18. Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., and Yuille, A. (2015). Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN). arXiv.
19. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
20. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and Tell: A Neural Image Caption Generator. Proceedings of the 2015 IEEE Conference on Computer Visionand Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
21. Long-Term Recurrent Convolutional Networks for Visual Recognition and Description10.1109/TPAMI.2016.2599174 ↗
22. Bahdanau, D., Cho, K., and Bengio, Y. (2016). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv.
23. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014;, January 25–29). Learning Phrase Representations Using RNN Encoder–Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference onEmpirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1179 ↗
24. Kalchbrenner, N., and Blunsom, P. (2013, January 18–21). Recurrent Continuous Translation Models. Proceedings of the 2013 Conference onEmpirical Methods in Natural Language Processing, Seattle, WA, USA.
25. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K.Q. (2014, January 8–13). Sequence to Sequence Learning with Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
26. Kiros, R., Salakhutdinov, R., and Zemel, R.S. (2014). Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models. arXiv.
27. Karpathy, A., and Fei-Fei, L. (2015, January 7–12). Deep Visual-Semantic Alignments for Generating Image Descriptions. Proceedings of the 2015 IEEE Conference on Computer Visionand Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298932 ↗
28. Ba, J., Mnih, V., and Kavukcuoglu, K. (2015). Multiple Object Recognition with Visual Attention. arXiv.
29. Mnih, V., Heess, N., Graves, A., and Kavukcuoglu, K. (2014, January 8–13). Recurrent Models of Visual Attention. Proceedings of the 27th International Conference on Neural Information Processing Systems—Volume 2, Montreal, QC, Canada.
30. Aligning Where to See and What to Tell: Image Captioning with Region-Based Attention and Scene-Specific Contexts10.1109/TPAMI.2016.2642953 ↗
31. Describing Multimedia Content Using Attention-based Encoder–Decoder Networks10.1109/TMM.2015.2477044 ↗
32. You, Q., Jin, H., Wang, Z., Fang, C., and Luo, J. (July, January 26). Image Captioning with Semantic Attention. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.503 ↗
33. Lu, J., Xiong, C., Parikh, D., and Socher, R. (2017, January 22–25). Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.345 ↗
34. Xu, K., Ba, J.L., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R.S., and Bengio, Y. (2015, January 6–11). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Proceedings of the 32nd International Conference on Machine Learning—Volume 37, Lille, France.
35. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 8–22). Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
36. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017, January 4–9). Attention Is All You Need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
37. Bloem, P. (2022, May 12). Transformers from Scratch|Peterbloem.Nl. Available online: http://peterbloem.nl/blog/transformers.
38. Li, X., Yin, X., Li, C., Hu, X., Zhang, P., Zhang, L., Wang, L., Hu, H., Dong, L., and Wei, F. (2020, January 23–28). Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58577-8_8 ↗
39. Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., and Gao, J. (2021, January 19–25). VinVL: Revisiting Visual Representations in Vision-Language Models. Proceedings of the Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00553 ↗
40. Hu, X., Yin, X., Lin, K., Wang, L., Zhang, L., Gao, J., and Liu, Z. (2021, January 2–9). VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training. Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, Virtual Event.
41. Hindawi—An Overview of Image Caption Generation Methods10.1155/2020/3062706 ↗
42. A Comprehensive Survey of Deep Learning for Image Captioning10.1145/3295748 ↗
43. Chaudhari, S., Mithal, V., Polatkan, G., and Ramanath, R. (2021). An Attentive Survey of Attention Models. arXiv.10.1145/3465055 ↗
44. Khan, S., Naseer, M., Hayat, M., Zamir, S.W., Khan, F.S., and Shah, M. (2021). Transformers in Vision: A Survey. arXiv.10.1145/3505244 ↗
45. From Show to Tell: A Survey on Deep Learning-based Image Captioning10.1109/TPAMI.2022.3148210 ↗
46. Services, A.W. (2023, August 15). Amazon Mechanical Turk. Available online: https://www.mturk.com/.
47. University, P. (2023, August 15). WordNet. Available online: https://wordnet.princeton.edu/.
48. WordNet: A Lexical Database for English10.1145/219717.219748 ↗
49. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). ImageNet: A Large-Scale Hierarchical Image Database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
50. Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014, January 6–12). Microsoft COCO: Common Objects in Context. Proceedings of the Computer Vision—ECCV, Zurich, Switzerland.10.1007/978-3-319-10602-1 ↗
51. The Open Images Dataset V410.1007/s11263-020-01316-z ↗
52. Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., and Wang, L. (2022, January 19–24). Scaling Up Vision-Language Pretraining for Image Captioning. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01745 ↗
53. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models from Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual Event.
54. Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., and Duerig, T. (2021, January 18–24). Scaling up Visual and Vision-Language Representation Learning with Noisy Text Supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual Event.
55. Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A.J., Padlewski, P., Salz, D., Goodman, S.A., Grycner, A., Mustafa, B., and Beyer, L. (2023, January 1–5). PaLI: A Jointly-Scaled Multilingual Language-Image Model. Proceedings of the International Conference on Learning Representations, Kigali, Rwanda.
56. Lab, S.V. (2018, March 02). ImageNet. Available online: http://image-net.org/.
57. Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., and Anderson, P. (November, January 27). Nocaps: Novel Object Captioning at Scale. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00904 ↗
58. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations10.1007/s11263-016-0981-7 ↗
59. Sharma, P., Ding, N., Goodman, S., and Soricut, R. (2018, January 15–20). Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1238 ↗
60. Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. (2021, January 19–25). Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training to Recognize Long-Tail Visual Concepts. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual Event.10.1109/CVPR46437.2021.00356 ↗
61. Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
62. From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference over Event Descriptions10.1162/tacl_a_00166 ↗
63. VQA: Visual Question Answering10.1007/s11263-016-0966-6 ↗
64. (2023, August 12). Visual Question Answering. Available online: http://visualqa.org.
65. Hudson, D.A., and Manning, C.D. (2019, January 16–20). GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00686 ↗
66. Ordonez, V., Kulkarni, G., and Berg, T.L. (2011, January 12–15). Im2Text: Describing Images Using 1 Million Captioned Photographs. Proceedings of the 24th International Conference on Neural Information Processing Systems, Granada, Spain.
67. Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., and Sun, J. (November, January 27). Objects365: A Large-Scale, High-Quality Dataset for Object Detection. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00852 ↗
68. Pavao, A., Guyon, I., Letournel, A.C., Baró, X., Escalante, H., Escalera, S., Thomas, T., and Xu, Z. (2022). CodaLab Competitions: An Open Source Platform to Organize Scientific Challenges, Université Paris-Saclay. Technical Report.
69. (2023, January 31). CodaLab-Microsoft COCO Image Captioning Challenge. Available online: https://competitions.codalab.org/competitions/3221#results.
70. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
71. Lin, C.Y. (2004, January 25–26). ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of the ACL-04 Workshop, Barcelona, Spain.
72. Banerjee, S., and Lavie, A. (2005, January 29). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, Michigan.
73. Vedantam, R., Zitnick, C.L., and Parikh, D. (2015, January 7–12). CIDEr: Consensus-based Image Description Evaluation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7299087 ↗
74. Leibe, B., Matas, J., Sebe, N., and Welling, M. (2016, January 8–16). SPICE: Semantic Propositional Image Caption Evaluation. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.10.1007/978-3-319-46454-1 ↗
75. Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollar, P., and Zitnick, C.L. (2015). Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv.
76. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2016). Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. arXiv.10.1609/aaai.v31i1.11231 ↗
77. Zhu, X., Li, L., Liu, J., Peng, H., and Niu, X. (2018). Captioning Transformer with Stacked Attention Modules. Appl. Sci., 8.10.3390/app8050739 ↗
78. Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. (2017, January 22–25). Aggregated Residual Transformations for Deep Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.634 ↗
79. Herdade, S., Kappeler, A., Boakye, K., and Soares, J. (2019, January 8–14). Image Captioning: Transforming Objects into Words. Proceedings of the Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
80. Hu, H., Gu, J., Zhang, Z., Dai, J., and Wei, Y. (2018, January 18–23). Relation Networks for Object Detection. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00378 ↗
81. Yang, X., Tang, K., Zhang, H., and Cai, J. (2019, January 16–20). Auto-Encoding Scene Graphs for Image Captioning. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01094 ↗
82. Yao, T., Pan, Y., Li, Y., and Mei, T. (2018, January 8–14). Exploring Visual Relationship for Image Captioning. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_42 ↗
83. Yao, T., Pan, Y., Li, Y., and Mei, T. (November, January 27). Hierarchy Parsing for Image Captioning. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00271 ↗
84. Huang, L., Wang, W., Chen, J., and Wei, X.Y. (November, January 27). Attention on Attention for Image Captioning. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00473 ↗
85. Li, G., Zhu, L., Liu, P., and Yang, Y. (November, January 27). Entangled Transformer for Image Captioning. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00902 ↗
86. Cornia, M., Stefanini, M., Baraldi, L., and Cucchiara, R. (2020). Meshed-Memory Transformer for Image Captioning. arXiv.10.1109/CVPR42600.2020.01059 ↗
87. Pan, Y., Yao, T., Li, Y., and Mei, T. (2020, January 13–19). X-Linear Attention Networks for Image Captioning. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.01098 ↗
88. Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., and Rohrbach, M. (2016, January 1–4). Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1044 ↗
89. Hu, J., Shen, L., and Sun, G. (2018, January 18–23). Squeeze-and-Excitation Networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00745 ↗
90. Ishikawa, H., Liu, C.L., Pajdla, T., and Shi, J. (December, January 30). Image Captioning Through Image Transformer. Proceedings of the Asian Conference on Computer Vision (ACCV), Kyoto, Japan.
91. Li, Y., Pan, Y., Yao, T., and Mei, T. (2022, January 19–24). Comprehending and Ordering Semantics for Image Captioning. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01746 ↗
92. Li, Y., Pan, Y., Chen, J., Yao, T., and Mei, T. (2021, January 20–24). X-Modaler: A Versatile and High-Performance Codebase for Cross-Modal Analytics. Proceedings of the 29th ACM International Conference on Multimedia, Chengdu, China.10.1145/3474085.3478331 ↗
93. Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey10.1109/TPAMI.2020.2992393 ↗
94. Self-Supervised Learning: Generative or Contrastive
95. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, Minneapolis, MN, USA.
96. “Cloze Procedure”: A New Tool for Measuring Readability
97. Lu, J., Batra, D., Parikh, D., and Lee, S. (2019, January 8–14). ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. Proceedings of the Annual Conference on Neural and Information Processing Systems, Vancouver, BC, Canada.
98. Inui, K., Jiang, J., Ng, V., and Wan, X. (2019, January 3–7). LXMERT: Learning Cross-Modality Encoder Representations from Transformers. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.
99. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations, Virtual Event, Austria.
100. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks