From Large Language Models to Large Multimodal Models: A Literature Review
Source abstract
Abstract
With the deepening of research on Large Language Models (LLMs), significant progress has been made in recent years on the development of Large Multimodal Models (LMMs), which are gradually moving toward Artificial General Intelligence. This paper aims to summarize the recent progress from LLMs to LMMs in a comprehensive and unified way. First, we start with LLMs and outline various conceptual frameworks and key techniques. Then, we focus on the architectural components, training strategies, fine-tuning guidance, and prompt engineering of LMMs, and present a taxonomy of the latest vision–language LMMs. Finally, we provide a summary of both LLMs and LMMs from a unified perspective, make an analysis of the development status of large-scale models in the view of globalization, and offer potential research directions for large-scale models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Applied Science, Shenzhen University, Shenzhen 518052, China
Department of Computer Science, George Mason University, Fairfax, VA 22030, USA
College of Big Data and Internet, Shenzhen Technology University, Shenzhen 518118, China
Funding
National Natural Science Foundation of China · Stable Support Projects for Shenzhen Higher Education Institutions · Natural Science Foundation of Top Talent of SZTU
References (100+)
Show source-provided references
1. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
2. Radford, A., and Narasimhan, K. (2024, March 29). Improving Language Understanding by Generative Pre-Training. Available online: https://api.semanticscholar.org/CorpusID:49313245.
3. Language Models are Unsupervised Multitask Learners
4. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language Models are Few-Shot Learners. arXiv.
5. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., and Altman, S. (2024). GPT-4 Technical Report. arXiv.
6. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., and Gehrmann, S. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv.
7. Anil, R., Dai, A.M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., and Chen, Z. (2023). PaLM 2 Technical Report. arXiv.
8. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv.
9. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv.
10. Zeng, W., Ren, X., Su, T., Wang, H., Liao, Y., Wang, Z., Jiang, X., Yang, Z., Wang, K., and Zhang, X. (2021). PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation. arXiv.
11. Ren, X., Zhou, P., Meng, X., Huang, X., Wang, Y., Wang, W., Li, P., Zhang, X., Podolskiy, A., and Arshinov, G. (2023). PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing. arXiv.
12. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv.
13. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. arXiv.10.1109/CVPR52688.2022.01042 ↗
14. Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. (2023). SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. arXiv.
15. A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges10.1109/ACCESS.2024.3365742 ↗
16. Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2024). A Comprehensive Overview of Large Language Models. arXiv.
17. Zhang, D., Yu, Y., Li, C., Dong, J., Su, D., Chu, C., and Yu, D. (2024). MM-LLMs: Recent Advances in MultiModal Large Language Models. arXiv.10.18653/v1/2024.findings-acl.738 ↗
18. Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. (2024). A Survey on Multimodal Large Language Models. arXiv.10.1093/nsr/nwae403 ↗
19. Lipton, Z.C., Berkowitz, J., and Elkan, C. (2015). A Critical Review of Recurrent Neural Networks for Sequence Learning. arXiv.
20. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
21. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2023). Attention Is All You Need. arXiv.
22. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
23. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv.
24. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P.J. (2023). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv.
25. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.W. (2019). Unified Language Model Pre-training for Natural Language Understanding and Generation. arXiv.
26. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-Efficient Transfer Learning for NLP. arXiv.
27. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv.
28. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv.
29. Li, X.L., and Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv.10.18653/v1/2021.acl-long.353 ↗
30. Schick, T., and Schütze, H. (2021). It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. arXiv.10.18653/v1/2021.naacl-main.185 ↗
31. Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. (2023). GPT Understands, Too. arXiv.10.1016/j.aiopen.2023.08.012 ↗
32. Lester, B., Al-Rfou, R., and Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv.10.18653/v1/2021.emnlp-main.243 ↗
33. Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., Li, L., and Sui, Z. (2023). A Survey on In-context Learning. arXiv.
34. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2023). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv.
35. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., and Nakano, R. (2021). Training Verifiers to Solve Math Word Problems. arXiv.
36. Zhang, Z., Gu, Y., Han, X., Chen, S., Xiao, C., Sun, Z., Yao, Y., Qi, F., Guan, J., and Ke, P. (2021). CPM-2: Large-scale Cost-effective Pre-trained Language Models. arXiv.10.1016/j.aiopen.2021.12.003 ↗
37. Zhang, Z., Han, X., Zhou, H., Ke, P., Gu, Y., Ye, D., Qin, Y., Su, Y., Ji, H., and Guan, J. (2020). CPM: A Large-scale Generative Chinese Pre-trained Language Model. arXiv.10.1016/j.aiopen.2021.07.001 ↗
38. Qin, Y., Lin, Y., Yi, J., Zhang, J., Han, X., Zhang, Z., Su, Y., Liu, Z., Li, P., and Sun, M. (2022). Knowledge Inheritance for Pre-trained Language Models. arXiv.10.18653/v1/2022.naacl-main.288 ↗
39. Barham, P., Chowdhery, A., Dean, J., Ghemawat, S., Hand, S., Hurt, D., Isard, M., Lim, H., Pang, R., and Roy, S. (2022). Pathways: Asynchronous Distributed Dataflow for ML. arXiv.
40. Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., and Young, S. (2022). Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv.
41. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L.A., Welbl, J., and Clark, A. (2022). Training Compute-Optimal Large Language Models. arXiv.
42. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., and Lin, X.V. (2022). OPT: Open Pre-trained Transformer Language Models. arXiv.
43. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., and Nabeshima, N. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv.
44. Baumgartner, J., Zannettou, S., Keegan, B., Squire, M., and Blackburn, J. (2020). The Pushshift Reddit Dataset. arXiv.10.1609/icwsm.v14i1.7347 ↗
45. Zhang, B., and Sennrich, R. (2019). Root Mean Square Layer Normalization. arXiv.
46. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. (2023). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv.10.1016/j.neucom.2023.127063 ↗
47. Xu, L., Zhang, X., and Dong, Q. (2020). CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model. arXiv.
48. WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models10.1016/j.aiopen.2021.06.001 ↗
49. Christopoulou, F., Lampouras, G., Gritta, M., Zhang, G., Guo, Y., Li, Z., Zhang, Q., Xiao, M., Shen, B., and Li, L. (2022). PanGu-Coder: Program Synthesis with Function-Level Language Modeling. arXiv.
50. Gousios, G. (2013, January 18–19). The GHTorent dataset and tool suite. Proceedings of the 2013 10th Working Conference on Mining Software Repositories (MSR), San Francisco, CA, USA.10.1109/MSR.2013.6624034 ↗
51. Tay, Y., Dehghani, M., Tran, V.Q., Garcia, X., Wei, J., Wang, X., Chung, H.W., Bahri, D., Schuster, T., and Zheng, S. (2023, January 1–5). UL2: Unifying Language Learning Paradigms. Proceedings of the Eleventh International Conference on Learning Representations, Kigali, Rwanda.
52. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv.10.18653/v1/2023.emnlp-main.298 ↗
53. Sanh, V., Webson, A., Raffel, C., Bach, S.H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T.L., and Raja, A. (2022). Multitask Prompted Training Enables Zero-Shot Task Generalization. arXiv.
54. (2024, March 29). Introducing ChatGPT. Available online: https://openai.com/blog/chatgpt.
55. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., and Saunders, W. (2022). WebGPT: Browser-assisted question-answering with human feedback. arXiv.
56. (2024, March 29). Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality|LMSYS Org. Available online: https://lmsys.org/blog/2023-03-30-vicuna/.
57. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T.B. (2024, March 29). Stanford Alpaca: An Instruction-Following LLaMA Model. Available online: https://github.com/tatsu-lab/stanford_alpaca.
58. Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., and Yu, L. (2023). LIMA: Less Is More for Alignment. arXiv.
59. Iyer, S., Lin, X.V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., and Koura, P.S. (2023). OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization. arXiv.
60. Li, J., Li, D., Savarese, S., and Hoi, S. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv.
61. Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. (2023). MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv.
62. Qin, J., Wu, J., Chen, W., Ren, Y., Li, H., Wu, H., Xiao, X., Wang, R., and Wen, S. (2024). DiffusionGPT: LLM-Driven Text-to-Image Generation System. arXiv.
63. Pan, X., Dong, L., Huang, S., Peng, Z., Chen, W., and Wei, F. (2023). Kosmos-G: Generating Images in Context with Multimodal Large Language Models. arXiv.
64. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. (2021). Masked Autoencoders Are Scalable Vision Learners. arXiv.10.1109/CVPR52688.2022.01553 ↗
65. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021). An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. arXiv.
66. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. arXiv.10.1109/ICCV48922.2021.00986 ↗
67. Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. (2022). EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. arXiv.10.1109/CVPR52729.2023.01855 ↗
68. Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. (2022). Reproducible scaling laws for contrastive language-image learning. arXiv.10.1109/CVPR52729.2023.00276 ↗
69. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., and El-Nouby, A. (2024). DINOv2: Learning Robust Visual Features without Supervision. arXiv.
70. Wu, Y., Chen, K., Zhang, T., Hui, Y., Nezhurina, M., Berg-Kirkpatrick, T., and Dubnov, S. (2024). Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation. arXiv.10.1109/ICASSP49357.2023.10095969 ↗
71. Chen, F., Han, M., Zhao, H., Zhang, Q., Shi, J., Xu, S., and Xu, B. (2023). X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages. arXiv.
72. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2022). Robust Speech Recognition via Large-Scale Weak Supervision. arXiv.
73. Hsu, W.N., Bolte, B., Tsai, Y.H.H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021). HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. arXiv.10.1109/TASLP.2021.3122291 ↗
74. Zhang, Y., Gong, K., Zhang, K., Li, H., Qiao, Y., Ouyang, W., and Yue, X. (2023). Meta-Transformer: A Unified Framework for Multimodal Learning. arXiv.
75. Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., and Misra, I. (2023). ImageBind: One Embedding Space To Bind Them All. arXiv.10.1109/CVPR52729.2023.01457 ↗
76. Zhu, B., Lin, B., Ning, M., Yan, Y., Cui, J., Wang, H., Pang, Y., Jiang, W., Zhang, J., and Li, Z. (2024). LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment. arXiv.
77. Jian, Y., Gao, C., and Vosoughi, S. (2023). Bootstrapping Vision-Language Learning with Decoupled Language Pre-training. arXiv.
78. Lu, J., Gan, R., Zhang, D., Wu, X., Wu, Z., Sun, R., Zhang, J., Zhang, P., and Song, Y. (2023). Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects. arXiv.
79. Koh, J.Y., Fried, D., and Salakhutdinov, R. (2023). Generating Images with Multimodal Language Models. arXiv.
80. Tian, C., Zhu, X., Xiong, Y., Wang, W., Chen, Z., Wang, W., Chen, Y., Lu, L., Lu, T., and Zhou, J. (2024). MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer. arXiv.
81. Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., and Lewis, M. (2022). CM3: A Causal Masked Multimodal Model of the Internet. arXiv.
82. Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M.D. (2023). AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. arXiv.
83. Liu, H., Tian, Q., Yuan, Y., Liu, X., Mei, X., Kong, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M.D. (2023). AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining. arXiv.10.1109/TASLP.2024.3399607 ↗
84. Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M., and Le, Q.V. (2022). Finetuned Language Models Are Zero-Shot Learners. arXiv.
85. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., and Ray, A. (2022). Training language models to follow instructions with human feedback. arXiv.
86. Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. (2023). Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic. arXiv.
87. Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., and Qiao, Y. (2024). VideoChat: Chat-Centric Video Understanding. arXiv.
88. Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., and Sun, T. (2024). LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding. arXiv.
89. Liu, X., Ji, K., Fu, Y., Tam, W.L., Du, Z., Yang, Z., and Tang, J. (2022). P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. arXiv.10.18653/v1/2022.acl-short.8 ↗
90. Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., and Duan, N. (2023). Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models. arXiv.
91. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., and Kalyan, A. (2022). Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. arXiv.
92. Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. (2023). Multimodal Chain-of-Thought Reasoning in Language Models. arXiv.
93. Ge, J., Luo, H., Qian, S., Gan, Y., Fu, J., and Zhang, S. (2023). Chain of Thought Prompt Tuning in Vision Language Models. arXiv.
94. Hu, W., Xu, Y., Li, Y., Li, W., Chen, Z., and Tu, Z. (2023). BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions. arXiv.10.1609/aaai.v38i3.27999 ↗
95. Zhao, L., Yu, E., Ge, Z., Yang, J., Wei, H., Zhou, H., Sun, J., Peng, Y., Dong, R., and Han, C. (2023). ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning. arXiv.
96. Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., and Song, X. (2023). CogVLM: Visual Expert for Pretrained Language Models. arXiv.
97. Chen, Y., Sikka, K., Cogswell, M., Ji, H., and Divakaran, A. (2023). DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback. arXiv.
98. (2024, March 29). Introducing IDEFICS: An Open Reproduction of State-of-the-Art Visual Langage Model. Available online: https://huggingface.co/blog/idefics.
99. Zhang, P., Dong, X., Wang, B., Cao, Y., Xu, C., Ouyang, L., Zhao, Z., Duan, H., Zhang, S., and Ding, S. (2023). InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition. arXiv.
100. Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Wei, X., Zhang, S., Duan, H., and Cao, M. (2024). InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model. arXiv.