A Survey of Robot Intelligence with Large Language Models
Source abstract
Abstract
Since the emergence of ChatGPT, research on large language models (LLMs) has actively progressed across various fields. LLMs, pre-trained on vast text datasets, have exhibited exceptional abilities in understanding natural language and planning tasks. These abilities of LLMs are promising in robotics. In general, traditional supervised learning-based robot intelligence systems have a significant lack of adaptability to dynamically changing environments. However, LLMs help a robot intelligence system to improve its generalization ability in dynamic and complex real-world environments. Indeed, findings from ongoing robotics studies indicate that LLMs can significantly improve robots’ behavior planning and execution capabilities. Additionally, vision-language models (VLMs), trained on extensive visual and linguistic data for the vision question answering (VQA) problem, excel at integrating computer vision with natural language processing. VLMs can comprehend visual contexts and execute actions through natural language. They also provide descriptions of scenes in natural language. Several studies have explored the enhancement of robot intelligence using multimodal data, including object recognition and description by VLMs, along with the execution of language-driven commands integrated with visual information. This review paper thoroughly investigates how foundation models such as LLMs and VLMs have been employed to boost robot intelligence. For clarity, the research areas are categorized into five topics: reward design in reinforcement learning, low-level control, high-level planning, manipulation, and scene understanding. This review also summarizes studies that show how foundation models, such as the Eureka model for automating reward function design in reinforcement learning, RT-2 for integrating visual data, language, and robot actions in vision-language-action models, and AutoRT for generating feasible tasks and executing robot behavior policies via LLMs, have improved robot intelligence.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Mechanical Engineering, Dong-A University, Busan 49315, Republic of Korea
School of Mechanical Engineering, Pukyong National University, Busan 48513, Republic of Korea
Funding
Ministry of Trade, Industry, and Energy (MOTIE, Korea) · Ministry of Education (MOE)
References (100+)
Show source-provided references
1. (2024, August 13). Hello GPT-4o. Available online: https://openai.com/index/hello-gpt-4o/.
2. ChatGPT for Robotics: Design Principles and Model Abilities10.1109/ACCESS.2024.3387941 ↗
3. Hu, Y., Xie, Q., Jain, V., Francis, J., Patrikar, J., Keetha, N., Kim, S., Xie, Y., Zhang, T., and Zhao, S. (2023). Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis. arXiv.
4. Xiao, X., Liu, J., Wang, Z., Zhou, Y., Qi, Y., Cheng, Q., He, B., and Jiang, S. (2023). Robot Learning in the Era of Foundation Models: A Survey. arXiv.10.2139/ssrn.4706193 ↗
5. Mao, Y., Ge, Y., Fan, Y., Xu, W., Mi, Y., Hu, Z., and Gao, Y. (2024). A Survey on LoRA of Large Language Models. arXiv.
6. Hunt, W., Ramchurn, S.D., and Soorati, M.D. (2024). A Survey of Language-Based Communication in Robotics. arXiv.
7. Learning Transferable Visual Models From Natural Language Supervision
8. Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., and Hsu, J. (2023, January 10–14). RT-1: Robotics Transformer for Real-World Control at Scale. Proceedings of the Robotics: Science and Systems 2023, Daegu, Republic of Korea.10.15607/RSS.2023.XIX.025 ↗
9. Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., and Finn, C. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv.
10. Ahn, M., Dwibedi, D., Finn, C., Arenas, M.G., Gopalakrishnan, K., Hausman, K., Ichter, B., Irpan, A., Joshi, N., and Julian, R. (2024). AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents. arXiv.
11. Ma, Y.J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. (2023). Eureka: Human-Level Reward Design via Coding Large Language Models. arXiv.
12. Ma, Y., Song, Z., Zhuang, Y., Hao, J., and King, I. (2024). A Survey on Vision-Language-Action Models for Embodied AI. arXiv.
13. Zhou, H., Yao, X., Meng, Y., Sun, S., Bing, Z., Huang, K., and Knoll, A. (2023). Language-Conditioned Learning for Robotic Manipulation: A Survey. arXiv.
14. Firoozi, R., Tucker, J., Tian, S., Majumdar, A., Sun, J., Liu, W., Zhu, Y., Song, S., Kapoor, A., and Hausman, K. (2023). Foundation Models in Robotics: Applications, Challenges, and the Future. arXiv.10.1177/02783649241281508 ↗
15. Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
16. Zhai, C. (2007). Statistical Language Models for Information Retrieval, Association for Computational Linguistics.10.1561/9781601981875 ↗
17. Introduction to the Special Issue on Statistical Language Modeling10.1145/1034780.1034781 ↗
18. Two Decdes of Statistical Language Modeling Where Do We Go Form Here? Where Do We Go from Here?10.1109/5.880083 ↗
19. Gondala, S., Verwimp, L., Pusateri, E., Tsagkias, M., and Van Gysel, C. (2021, January 6–11). Error-Driven Pruning of Language Models for Virtual Assistants. Proceedings of the ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9415035 ↗
20. Statistical Language Modeling for Information Retrieval10.1002/aris.1440390108 ↗
21. Thede, S.M., and Harper, M.P. (1999, January 20–26). A Second-Order Hidden Markov Model for Part-of-Speech Tagging. Proceedings of the Annual Meeting of the Association for Computational Linguistics, College Park, MD, USA.10.3115/1034678.1034712 ↗
22. A Tree-Based Statistical Language Model for Natural Language Speech Recognition10.1109/29.32278 ↗
23. Brants, T., Popat, A.C., Xu, P., Och, F.J., and Dean, J. (2007, January 28–30). Large Language Models in Machine Translation. Proceedings of the EMNLP-CoNLL 2007-Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Prague, Czech Republic.
24. Catalytic 3D Polymerization of C6010.1080/1536383X.2018.1448388 ↗
25. Mikolov, T., Karafiát, M., Burget, L., Jan, C., and Khudanpur, S. (2010, January 26–30). Recurrent Neural Network Based Language Model. Proceedings of the 11th Annual Conference of the International Speech Communication Association, INTERSPEECH 2010, Chiba, Japan.10.21437/Interspeech.2010-343 ↗
26. Kombrink, S., Mikolov, T., Karafiát, M., and Burget, L. (2011, January 27–31). Recurrent Neural Network Based Language Modeling in Meeting Recognition. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, Florence, Italy.10.21437/Interspeech.2011-720 ↗
27. Natural Language Processing (Almost) from Scratch
28. Distributed Representations Ofwords and Phrases and Their Compositionality
29. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013, January 2–4). Efficient Estimation of Word Representations in Vector Space. Proceedings of the 1st International Conference on Learning Representations, ICLR 2013-Workshop Track Proceedings, Scottsdale, AZ, USA.
30. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018, January 14). Deep Contextualized Word Representations. Proceedings of the NAACL HLT 2018—2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA.10.18653/v1/N18-1202 ↗
31. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 10). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the NAACL HLT 2019—2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
32. Attention Is All You Need
33. BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension10.18653/v1/2020.acl-main.703 ↗
34. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
35. Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., and Brunskill, E. (2021). Language Models Are Unsupervised Multitask Learners. arXiv.
36. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
37. Sanh, V., Webson, A., Raffel, C., Bach, S.H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Le Scao, T., and Raja, A. (2022, January 25–29). Multitask Prompted Training Enables Zero-Shot Task Generalization. Proceedings of the ICLR 2022—10th International Conference on Learning Representations, Virtual.
38. What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
39. Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv.
40. Talking about Large Language Models10.1145/3624724 ↗
41. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
42. Training Compute-Optimal Large Language Models
43. Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. (2022). Galactica: A Large Language Model for Science. arXiv.
44. T-Model Structures10.4310/HHA.2007.v9.n1.a16 ↗
45. OLMo: Accelerating the Science of Language Models
46. Lozhkov, A., Li, R., Allal, L.B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., and Wei, Y. (2024). StarCoder 2 and The Stack v2: The Next Generation. arXiv.
47. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
48. (2024, August 13). The Claude 3 Model Family: Opus, Sonnet, Haiku. Available online: https://api.semanticscholar.org/CorpusID:268232499.
49. Language Models Are Few-Shot Learners
50. Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., and Chu, P. (2024). InternLM2 Technical Report. arXiv.
51. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., and Brockman, G. (2021). Evaluating Large Language Models Trained on Code. arXiv.
52. Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedigos, I., Safahi, E., Meirom, S., Belinkov, Y., and Shalev-Shwartz, S. (2024). Jamba: A Hybrid Transformer-Mamba Language Model. arXiv.
53. Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M., and Le, Q.V. (2022, January 25–29). Finetuned Language Models Are Zero-Shot Learners. Proceedings of the ICLR 2022—10th International Conference on Learning Representations, Virtual.
54. Pinnaparaju, N., Adithyan, R., Phung, D., Tow, J., Baicoianu, J., Datta, A., Zhuravinskyi, M., Mahan, D., Bellagente, M., and Riquelme, C. (2024). Stable Code Technical Report. arXiv.
55. Yoo, K.M., Han, J., In, S., Jeon, H., Jeong, J., Kang, J., Kim, H., Kim, K.-M., Kim, M., and Kim, S. (2024). HyperCLOVA X Technical Report. arXiv.
56. Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., and Young, S. (2021). Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv.
57. (2024, August 13). Grok-1.5 Vision Preview. Available online: https://x.ai/blog/grok-1.5v.
58. Training Language Models to Follow Instructions with Human Feedback
59. (2024, August 13). Vallabh Shrimangale Introducing Meta Llama 3: The Most Capable Openly Available LLM to Date. Available online: https://medium.com/@shrimangalevallabh789/introducing-meta-llama-3-the-most-capable-openly-available-llm-to-date-12de163151e1.
60. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., and Gehrmann, S. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv.
61. Abdin, M., Jacobs, S.A., Awan, A.A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., and Bao, J. (2024). Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv.
62. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., and Lin, X.V. (2022). OPT: Open Pre-Trained Transformer Language Models. arXiv.
63. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv.
64. (2024, August 13). Claude 3.5 Sonnet. Available online: https://www.anthropic.com/news/claude-3-5-sonnet.
65. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. (2024, August 13). Alpaca: A Strong, Replicable Instruction-Following Model. Available online: https://crfm.stanford.edu/2023/03/13/alpaca.html.
66. (2024, August 13). GPT-4o Mini: Advancing Cost-Efficient Intelligence. Available online: https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/.
67. Malartic, Q., Chowdhury, N.R., Cojocaru, R., Farooq, M., Campesan, G., Djilali, Y.A.D., Narayan, S., Singh, A., Velikanov, M., and Boussaha, B.E.A. (2024, August 13). Falcon2-11B Technical Report. Available online: https://huggingface.co/tiiuae/falcon-11B.
68. Li, R., Allal, L.B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., and Chim, J. (2023). StarCoder: May the Source Be with You!. arXiv.
69. (2024, August 13). Introducing Llama 3.1: Our Most Capable Models to Date. Available online: https://ai.meta.com/blog/meta-llama-3-1/.
70. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv.
71. (2024, August 13). Mistral AI Mistral Large. Available online: https://mistral.ai/news/mistral-large/?utm_source=www.turingpost.com&utm_medium=referral&utm_campaign=the-ultimate-guide-to-llm-benchmarks-evaluating-language-model-capabilities.
72. Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., and Yan, D. (2023). Baichuan 2: Open Large-Scale Language Models. arXiv.
73. Team, G., and Deepmind, G. (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv.
74. Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv.
75. An, S., Bae, K., Choi, E., Choi, S.J., Choi, Y., Hong, S., Hong, Y., Hwang, J., Jeon, H., and Gerrard, J.J. (2024). EXAONE 3.0 7.8B Instruction Tuned Language Model. arXiv.
76. Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., and Li, Y.K. (2024). DeepSeek-Coder: When the Large Language Model Meets Programming—The Rise of Code Intelligence. arXiv.
77. (2024, August 13). Grok-2 Beta Release. Available online: https://x.ai/blog/grok-2.
78. End-to-End Object Detection with Transformers
79. Liu, H., Li, C., Wu, Q., and Lee, Y.J. (2024, August 13). Visual Instruction Tuning. Available online: https://github.com/haotian-liu/LLaVA.
80. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2021). Training Data-Efficient Image Transformers & Distillation through Attention. arXiv.
81. Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. (2024, August 13). MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. Available online: http://arxiv.org/abs/2304.10592.
82. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-Shot Text-to-Image Generation. arXiv.
83. (2024, August 13). ChatGPT-4 System Card. Available online: https://cdn.openai.com/papers/gpt-4-system-card.pdf.
84. Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M., Liu, C., and Yuan, L. (2023). Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. arXiv.10.1109/CVPR52733.2024.00461 ↗
85. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2024, August 13). Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. Available online: https://github.com/microsoft/Swin-Transformer.
86. Bar-Tal, O., Chefer, H., Tov, O., Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Liu, G., and Raj, A. (2024). Lumiere: A Space-Time Diffusion Model for Video Generation. arXiv.
87. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
88. (2024, August 13). Adept Fuyu-Heavy: A New Multimodal Model. Available online: https://www.adept.ai/blog/adept-fuyu-heavy.
89. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An Image Is Worth 16X16 Words: Transformers for Image Recognition At Scale. Proceedings of the ICLR 2021—9th International Conference on Learning Representations, Virtual Event, Austria.
90. Gemini Team, Georgiev, P., Lei, V.I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., and Wang, S. (2024). Gemini 1.5: Unlocking Multimodal Understanding across Millions of Tokens of Context. arXiv.
91. Bao, H., Dong, L., Piao, S., and Wei, F. (2022, January 25–29). Beit: Bert Pre-Training of Image Transformers. Proceedings of the ICLR 2022—10th International Conference on Learning Representations, Virtual.
92. Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Zhang, S., Duan, H., Zhang, W., and Li, Y. (2024). InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD. arXiv.
93. He, K., Chen, X., Xie, S., Li, Y., Dollar, P., and Girshick, R. (2022, January 18–24). Masked Autoencoders Are Scalable Vision Learners. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01553 ↗
94. (2018, August 13). Introducing Idefics2: A Powerful 8B Vision-Language Model for the Community. Available online: https://huggingface.co/blog/idefics2.
95. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2024, August 13). High-Resolution Image Synthesis with Latent Diffusion Models. Available online: https://github.com/CompVis/latent-diffusion.
96. Laurençon, H., Tronchon, L., Cord, M., and Sanh, V. (2024). What Matters When Building Vision-Language Models?. arXiv.
97. Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. (2022). R3M: A Universal Visual Representation for Robot Manipulation. arXiv.
98. Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foundation Models. arXiv.
99. Flamingo: A Visual Language Model for Few-Shot Learning
100. Li, J., Li, D., Savarese, S., and Hoi, S. (2024, August 13). BLIP-2: Bootstrapping Language-Image Pre-Training with Frozen Image Encoders and Large Language Models. Available online: https://github.com/salesforce/LAVIS/tree/main/projects/blip2.