All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks
Source abstract
Abstract
Large Language Models (LLMs), such as ChatGPT, encounter ‘jailbreak’ challenges, wherein safeguards are circumvented to generate ethically harmful prompts. This study introduces a straightforward black-box method for efficiently crafting jailbreak prompts that bypass LLM defenses. Our technique iteratively transforms harmful prompts into benign expressions directly utilizing the target LLM, predicated on the hypothesis that LLMs can autonomously generate expressions that evade safeguards. Through experiments conducted with ChatGPT (GPT-3.5 and GPT-4) and Gemini-Pro, our method consistently achieved an attack success rate exceeding 80% within an average of five iterations for forbidden questions and proved to be robust against model updates. The jailbreak prompts generated were not only naturally worded and succinct, but also challenging to defend against. These findings suggest that the creation of effective jailbreak prompts is less complex than previously believed, underscoring the heightened risk posed by black-box jailbreak attacks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Adversarial Robustness in Machine Learning · Advanced Malware Detection Techniques · Digital and Cyber Forensics
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Bioscience and Bioinformatics, Kyushu Institute of Technology Iizuka, Fukuoka 820-8502, Japan
Funding
JSPS KAKENHI
References (31)
Show source-provided references
1. (2024, March 18). OpenAI. Introducing ChatGPT. Available online: https://openai.com/blog/chatgpt.
2. Fraiwan, M., and Khasawneh, N. (2023). A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directions. arXiv.
3. Sallam, M. (2023). ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare, 11.10.3390/healthcare11060887 ↗
4. Large language models in medicine10.1038/s41591-023-02448-8 ↗
5. Revisiting the political biases of ChatGPT10.3389/frai.2023.1232003 ↗
6. The Moral Machine Experiment on Large Language Models10.1098/rsos.231393 ↗
7. Language models are few-shot learners
8. Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., and Narasimhan, K. (2023). Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv.10.18653/v1/2023.findings-emnlp.88 ↗
9. Training language models to follow instructions with human feedback
10. Markov, T., Zhang, C., Agarwal, S., Nekoul, F.E., Lee, T., Adler, S., Jiang, A., and Weng, L. (2023, January 7–14). A holistic approach to undesired content detection in the real world. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.10.1609/aaai.v37i12.26752 ↗
11. Carlini, N., Nasr, M., Choquette-Choo, C.A., Jagielski, M., Gao, I., Awadalla, A., Koh, P.W., Ippolito, D., Lee, K., and Tramer, F. (2023). Are aligned neural networks adversarially aligned?. arXiv.
12. Chain-of-thought prompting elicits reasoning in large language models
13. Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y. (2023). “Do Anything Now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv.
14. (2024, March 18). coolaj86—Chat GPT “DAN” (and Other “Jailbreaks”). Available online: https://gist.github.com/coolaj86/6f4f7b30129b0251f61fa7baaa881516.
15. Zhu, S., Zhang, R., An, B., Wu, G., Barrow, J., Wang, Z., Huang, F., Nenkova, A., and Sun, T. (2023). AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models. arXiv.
16. Zou, A., Wang, Z., Kolter, J.Z., and Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models. arXiv.
17. Robey, A., Wong, E., Hassani, H., and Pappas, G.J. (2023). SmoothLLM: Defending large language models against jailbreaking attacks. arXiv.
18. Defending ChatGPT against jailbreak attack via self-reminders10.1038/s42256-023-00765-8 ↗
19. Alon, G., and Kamfonas, M. (2023). Detecting language model attacks with perplexity. arXiv.
20. Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P.Y., Goldblum, M., Saha, A., Geiping, J., and Goldstein, T. (2023). Baseline defenses for adversarial attacks against aligned language models. arXiv.
21. Lapid, R., Langberg, R., and Sipper, M. (2023). Open sesame! universal black box jailbreaking of large language models. arXiv.
22. Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G.J., and Wong, E. (2023). Jailbreaking black box large language models in twenty queries. arXiv.
23. Perez, F., and Ribeiro, I. (2022). Ignore previous prompt: Attack techniques for language models. arXiv.
24. Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., and Liu, Y. (2023). Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv.
25. Rao, A., Vashistha, S., Naik, A., Aditya, S., and Choudhury, M. (2023). Tricking LLMs into Disobedience: Understanding, Analyzing, and Preventing Jailbreaks. arXiv.
26. Shayegani, E., Mamun, M.A.A., Fu, Y., Zaree, P., Dong, Y., and Abu-Ghazaleh, N. (2023). Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv.
27. Adversarial attacks on deep-learning models in natural language processing: A survey
28. (2024, March 18). OpenAI. GPT-4. Available online: https://openai.com/research/gpt-4.
29. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., and Hauth, A. (2023). Gemini: A family of highly capable multimodal models. arXiv.
30. Yang, Y., Hui, B., Yuan, H., Gong, N., and Cao, Y. (2024, January 20–22). SneakyPrompt: Jailbreaking Text-to-image Generative Models. Proceedings of the 2024 IEEE Symposium on Security and Privacy (SP), Los Alamitos, CA, USA.10.1109/SP54263.2024.00123 ↗
31. Liu, X., Xu, N., Chen, M., and Xiao, C. (2024, January 7–11). Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. Proceedings of the The Twelfth International Conference on Learning Representations, Vienna, Austria.