The Impact of Prompting Techniques on the Security of the LLMs and the Systems to Which They Belong
Source abstract
Abstract
Large language models have demonstrated impressive capabilities. The recent research conducted in the field of prompt engineering showed that their base performance is just a glimpse of their full abilities. Enhanced with auxiliary tools and provided with examples of how to solve the tasks, their adoption into our applications seems trivial. In this context, we ask an uncomfortable question. Are the models secure enough to be adopted in our systems, or do they represent Trojan horses? The idea of prompt injection and jailbreak attacks does not seem to bother the adopters too much. Even though there are a lot of studies that look into the benefits of the prompting techniques, none address their possible downside in regard to the security. We want take a step further and investigate the impact of the most popular prompting techniques on this aspect of large language models and implicitly the systems to which they belong. Using three of the most deployed GPT models to date, we conducted a few of the most popular attacks in different setup scenarios and demonstrate that prompting techniques can have a negative impact on the security of the LLMs. More than that, they also expose other system components that otherwise would have been less exposed. In the end, we try to come up with possible solutions and present future research perspectives.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Law, AI, and Intellectual Property
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Computer Science, Alexandru Ioan Cuza University, 700483 Iasi, Romania
References (37)
Show source-provided references
1. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language models are few-shot learners. arXiv.
2. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. arXiv.
3. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. (2023). Pal: Program-aided language models. arXiv.
4. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). React: Synergizing reasoning and acting in language models. arXiv.
5. Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., Zhang, T., Liu, Y., Wang, H., and Zheng, Y. (2024). Prompt injection attack against llm-integrated applications. arXiv.
6. Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., Wang, K., and Liu, Y. (2024). Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv.10.1145/3663530.3665021 ↗
7. Perez, F., and Ribeiro, I. (2022). Ignore previous prompt: Attack techniques for language models. arXiv.
8. Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., and Fritz, M. (2023). Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. arXiv.10.1145/3605764.3623985 ↗
9. OWASP (2024, May 27). Owasp Top 10 for Large Language Model Applications. Available online: https://owasp.org/www-project-top-10-for-large-language-model-applications/.
10. (2024, May 16). N-Gram Language Models. Available online: https://web.stanford.edu/~jurafsky/slp3/.
11. An introduction to hidden markov models10.1109/MASSP.1986.1165342 ↗
12. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2023). Attention is all you need. arXiv.
13. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. (2017). Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv.10.18653/v1/P17-1015 ↗
14. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., and Nakano, R. (2021). Training verifiers to solve math word problems. arXiv.
15. Patel, A., Bhattamishra, S., and Goyal, N. (2021). Are nlp models really able to solve simple math word problems?. arXiv.10.18653/v1/2021.naacl-main.168 ↗
16. Miao, S., Liang, C.-C., and Su, K.-Y. (2020, January 5–10). A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Available online: https://aclanthology.org/2020.acl-main.92.10.18653/v1/2020.acl-main.92 ↗
17. Koncel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H. (2016, January 12–17). MAWPS: A Math Word Problem Repository. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA. Available online: https://aclanthology.org/N16-1136.10.18653/v1/N16-1136 ↗
18. Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. (2021). Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. arXiv.10.1162/tacl_a_00370 ↗
19. Nye, M., Andreassen, A., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., and Luan, D. (2021). Show your work: Scratchpads for intermediate computation with language models. arXiv.
20. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., and Le, Q. (2023). Least-to-most prompting enables complex reasoning in large language models. arXiv.
21. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., and Brockman, G. (2021). Evaluating large language models trained on code. arXiv.
22. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H.W., Chowdhery, A., Le, Q.V., Chi, E., and Zhou, D. (2022). Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv.10.18653/v1/2023.findings-acl.824 ↗
23. Madaan, A., and Yazdanbakhsh, A. (2022). Text and patterns: For effective chain of thought, it takes two to tango. arXiv.
24. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W.W., Salakhutdinov, R., and Manning, C.D. (2018). Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv.10.18653/v1/D18-1259 ↗
25. Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. (2021). Alfworld: Aligning text and embodied environments for interactive learning. arXiv.
26. Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mittal, A. (2018). Fever: A large-scale dataset for fact extraction and verification. arXiv.10.18653/v1/N18-1074 ↗
27. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., and Saunders, W. (2022). Webgpt: Browser-assisted question-answering with human feedback. arXiv.
28. Yao, S., Chen, H., Yang, J., and Narasimhan, K. (2023). Webshop: Towards scalable real-world web interaction with grounded language agents. arXiv.
29. Markov, T., Zhang, C., Agarwal, S., Eloundou, T., Lee, T., Adler, S., Jiang, A., and Weng, L. (2023). A holistic approach to undesired content detection in the real world. arXiv.10.1609/aaai.v37i12.26752 ↗
30. (2024, May 16). Langchain. Available online: https://python.langchain.com/v0.2/docs/introduction/.
31. Das, B.C., Amini, M.H., and Wu, Y. (2024). Security and privacy challenges of large language models: A survey. arXiv.
32. Lee, K. (2024, May 16). ChatGPT_DAN, Feb 2023. Available online: https://github.com/0xk1h0/ChatGPT_DAN.10.59653/jimat.v1i01.161 ↗
33. Apruzzese, G., Anderson, H.S., Dambra, S., Freeman, D., Pierazzi, F., and Roundy, K.A. (2022). “real attackers don’t compute gradients”: Bridging the gap between adversarial ml research and practice. arXi.10.1109/SaTML54575.2023.00031 ↗
34. WithSecure Labs (2024, June 18). Llm-Agent Prompt Injection. Available online: https://labs.withsecure.com/publications/llm-agent-prompt-injection.
35. Yotam, W., Noam, W., Oshri, A., Yoav, L., and Amnon, S. (2023). Fundamental Limitations of Alignment in Large Language Models. arXiv.
36. Irimia, C. (2022, January 2–4). Decentralized infrastructure for digital notarizing, signing and sharing files securely using Blockchain. Proceedings of the Symposium on Logic and Artificial Intelligence, Lefkosa, Cyprus.
37. Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. (2020). Alfred: A benchmark for interpreting grounded instructions for everyday tasks. arXiv.10.1109/CVPR42600.2020.01075 ↗