Optimizing Large Language Models: A Deep Dive into Effective Prompt Engineering Techniques
Source abstract
Abstract
Recent advancements in Natural Language Processing (NLP) technologies have been driven at an unprecedented pace by the development of Large Language Models (LLMs). However, challenges remain, such as generating responses that are misaligned with the intent of the question or producing incorrect answers. This paper analyzes various Prompt Engineering techniques for large-scale language models and identifies methods that can optimize response performance across different datasets without the need for extensive retraining or fine-tuning. In particular, we examine prominent Prompt Engineering techniques including In-Context Learning (ICL), Chain of Thought (CoT), Retrieval-Augmented Generation (RAG), Step-by-Step Reasoning (SSR), and Tree of Thought (ToT), and we apply these techniques to leading LLMs such as Gemma2, LlaMA3, and Mistral. The performance of these models was evaluated using the AI2 Reasoning Challenge (ARC), HellaSwag, Massive Multitask Language Understanding (MMLU), TruthfulQA, Winogrande, and Grade School Math (GSM8k) datasets across metrics such as BLEU, ROUGE, METEOR, BLEURT, and BERTScore. The experimental results indicate that the most suitable Prompt Engineering technique can vary depending on the characteristics of each dataset. Specifically, for datasets emphasizing mathematical and logical reasoning, Prompt Engineering strategies centered around CoT, SSR, and ToT were found to be advantageous. For datasets focusing on natural language understanding, ICL-centric strategies were more effective, while RAG-based strategies were beneficial for datasets where factual accuracy is crucial. However, it was also observed that the optimal combination of Prompt Engineering techniques could differ depending on the specific LLM, indicating that fine-tuning the Prompt Engineering approach to the model and dataset is essential for achieving the best performance. The findings indicate that as LLMs become more advanced, their reliance on Prompt Engineering (PE) techniques diminishes, yet the magnitude of their performance improvement when PE strategies are applied increases. Furthermore, these advanced models tend to depend less on ICL techniques while exhibiting a greater reliance on RAG strategies. It is also evident that implementing RAG with PE-based preprocessing yields superior performance enhancements compared to the mere application of RAG on raw data.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Speech and dialogue systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Metabiohealth, Sungkyunkwan University, Suwon 16419, Republic of Korea
Korea Electronics Technology Institute, Seongnam-si 13488, Republic of Korea
Department of Smart Automotive, Soonchunhyang University, Asan-si 31538, Republic of Korea
Funding
Ministry of Trade, Industry and Energy
References (45)
Show source-provided references
1. Rula, A., and D’Souza, J. (2023, January 5–7). Procedural Text Mining with Large Language Models. Proceedings of the 12th Knowledge Capture Conference 2023, Pensacola, FL, USA.10.1145/3587259.3627572 ↗
2. Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., and Millican, K. (2023). Gemini: A Family of Highly Capable Multimodal Models. arXiv.
3. Ye, J., Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., Cui, Y., Zhou, Z., Gong, C., and Shen, Y. (2023). A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models. arXiv.
4. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). GPT-4 Technical Report. arXiv.
5. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., and Gehrmann, S. (2023). PaLM: Scaling Language Modeling with Pathways. arXiv.
6. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1.
7. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention Is All You Need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
8. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv.
9. Meta (2025, January 23). Llama3. Meta AI Blog. Available online: https://ai.meta.com/blog/meta-llama-3/.
10. (2025, January 23). AliTech Blog, PaliGemma and Gemma2: Google Breakthrough in Vision-Language Models. Available online: https://alitech.io/blog/paligemma-and-gemma-2-google-breakthrough/.
11. Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.D.L., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv.
12. Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., Casas, D.D.L., Hanna, E.B., and Bressand, F. (2024). Mixtral of Experts. arXiv.
13. Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., Li, L., and Sui, Z. (2022). A Survey on In-context Learning. arXiv.
14. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., and Ray, A. (2022). Training language models to follow instructions with human feedback. arXiv.
15. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2023). Let us Verify Step by Step. arXiv.
16. Liu, F., and Lapata, M. (2019). Text Summarization with Pretrained Encoders. arXiv.10.18653/v1/D19-1387 ↗
17. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. (2024, January 10–15). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
18. Reynolds, L., and McDonell, K. (2021). Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. arXiv.10.1145/3411763.3451760 ↗
19. Gu, Y., Dong, L., Wei, F., and Huang, M. (2023, January 9–14). Pre-Training to Learn in Context. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada.10.18653/v1/2023.acl-long.267 ↗
20. Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. (2022, January 10–15). MetaICL: Learning to learn in context. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA, Online.10.18653/v1/2022.naacl-main.201 ↗
21. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. (2022, January 27). What makes good in-context examples for GPT-3?. Proceedings of the Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures (DeeLIO@ACL 2022), Dublin, Ireland.10.18653/v1/2022.deelio-1.10 ↗
22. Kim, H.J., Cho, H., Kim, J., Kim, T., Yoo, K.M., and Lee, S. (2022). Self-generated in-context learning: Leveraging autoregressive language models as a demonstration generator. arXiv.
23. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., and Zhou, D. (2022). Chain of Thought Prompting Elicits Reasoning in Large Language Models. arXiv.
24. Tonmoy, S.M., Zaman, S.M., Jain, V., Rani, A., Rawte, V., Chadha, A., and Das, A. (2024). A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models. arXiv.
25. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.T., and Rocktäschel, T. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv.
26. Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., and Chen, W. (2023). Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv.
27. Varshney, N., Yao, W., Zhang, H., Chen, J., and Yu, D. (2023). A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv.
28. Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A.T., Fan, Y., Zhao, V.Y., Lao, N., Lee, H., and Juan, D.C. (2023, January 9–14). Rarr: Researching and revising what language models say, using language models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada.10.18653/v1/2023.acl-long.910 ↗
29. Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., and Weston, J. (2023). Chain-of-verification reduces hallucination in large language models. arXiv.10.18653/v1/2024.findings-acl.212 ↗
30. Amatriain, X. (2024). Prompt Design and Engineering: Introduction and Advanced Methods. arXiv.
31. Chen, B., Zhang, Z., Langrené, N., and Zhu, S. (2023). Unleashing the potential of prompt engineering in Large Language Models: A comprehensive review. arXiv.
32. Singh, A., Ehtesham, A., Gupta, G.K., Chatta, N.K., Kumar, S., and Khoei, T.T. (2024). Exploring Prompt Engineering: A Systematic Review with SWOT Analysis. arXiv.
33. Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. (2021). Roformer: Enhanced transformer with rotary position embedding. arXiv.
34. Shazeer, N. (2020). GLU variants improve transformer. arXiv.
35. Steele, D., and Specia, L. (2018, January 1–6). Vis-Eval Metric Viewer: A Visualisation Tool for Inspecting and Evaluating Metric Scores of Machine Translation Output. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, New Orleans, LA, USA.10.18653/v1/N18-5015 ↗
36. Indowordnet’s help in Indian language machine translation10.1007/s00146-019-00907-w ↗
37. Popović, M., and Ney, H. (2009, January 30–31). Syntax-Oriented Evaluation Measures for Machine Translation Output. Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece.10.3115/1626431.1626435 ↗
38. Sellam, T., Das, D., and Parikh, A.P. (2020). BLEURT: Learning Robust Metrics for Text Generation. arXiv.10.18653/v1/2020.acl-main.704 ↗
39. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2019). BERTScore: Evaluating Text Generation with BERT. arXiv.
40. Boratko, M., Padigela, H., Mikkilineni, D., Yuvraj, P., Das, R., McCallum, A., Chang, M., Fokoue-Nkoutche, A., Kapanipathi, P., and Mattei, N. (2018). A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset. arXiv.10.18653/v1/W18-2607 ↗
41. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?. arXiv.10.18653/v1/P19-1472 ↗
42. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020). Measuring Massive Multitask Language Understanding. arXiv.
43. Lin, S., Hilton, J., and Evans, O. (2021). TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv.10.18653/v1/2022.acl-long.229 ↗
44. Sakaguchi, K., Bras, R.L., Bhagavatula, C., and Choi, Y. (2021). Winogrande: An Adversarial Winograd Schema Challenge at Scale. arXiv.10.1145/3474381 ↗
45. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., and Nakano, R. (2021). Training Verifiers to Solve Math Word Problems. arXiv.