Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback
Source abstract
Abstract
Evaluating query-passage relevance is a crucial task in information retrieval (IR), where the performance of large language models (LLMs) greatly depends on the quality of prompts. Current prompt optimization methods typically require multiple candidate generations or iterative refinements, resulting in significant computational overhead and limited practical applicability. In this paper, we propose a novel prompt optimization method that leverages LLM-based confusion matrix feedback to efficiently optimize prompts for the relevance evaluation task. Unlike previous approaches, our method systematically analyzes LLM predictions—both correct and incorrect—using a confusion matrix, enabling prompt refinement through a single-step update. Our experiments in realistic IR scenarios demonstrate that our method achieves competitive or superior performance compared to existing methods while drastically reducing computational costs, highlighting its potential as a practical and scalable solution.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Text Analysis Techniques · Topic Modeling · Sentiment Analysis and Opinion Mining
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Division of AI Data Convergence, Hankuk University of Foreign Studies, Yongin 17035, Republic of Korea
Funding
Hankuk University of Foreign Studies
References (31)
Show source-provided references
1. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., and Nakano, R. (2021). Training verifiers to solve math word problems. arXiv.
2. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., and Metzler, D. (2022). Emergent abilities of large language models. arXiv.
3. Sanh, V., Webson, A., Raffel, C., Bach, S.H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T.L., and Raja, A. (2021). Multitask prompted training enables zero-shot task generalization. arXiv.
4. Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. (2021). Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv.10.18653/v1/2022.acl-long.556 ↗
5. Thomas, P., Spielman, S., Craswell, N., and Mitra, B. (2023). Large language models can accurately predict searcher preferences. arXiv.10.1145/3626772.3657707 ↗
6. Reynolds, L., and McDonell, K. (2021, January 8–13). Prompt programming for large language models: Beyond the few-shot paradigm. Proceedings of the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.10.1145/3411763.3451760 ↗
7. Gao, T., Fisch, A., and Chen, D. (2020). Making pre-trained language models better few-shot learners. arXiv.10.18653/v1/2021.acl-long.295 ↗
8. Webson, A., and Pavlick, E. (2021). Do prompt-based models really understand the meaning of their prompts?. arXiv.10.18653/v1/2022.naacl-main.167 ↗
9. Zhou, Y., Muresanu, A.I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. (2022, January 25–29). Large Language Models are Human-Level Prompt Engineers. Proceedings of the Eleventh International Conference on Learning Representations, Virtual.
10. Pryzant, R., Iter, D., Li, J., Lee, Y.T., Zhu, C., and Zeng, M. (2023). Automatic prompt optimization with" gradient descent" and beam search. arXiv.10.18653/v1/2023.emnlp-main.494 ↗
11. Li, M., Wang, W., Feng, F., Cao, Y., Zhang, J., and Chua, T.S. (2023). Robust prompt optimization for large language models against distribution shifts. arXiv.10.18653/v1/2023.emnlp-main.95 ↗
12. Sun, H., Li, X., Xu, Y., Homma, Y., Cao, Q., Wu, M., Jiao, J., and Charles, D. (2023). Autohint: Automatic prompt optimization with hint generation. arXiv.
13. Ye, Q., Axmed, M., Pryzant, R., and Khani, F. (2023). Prompt engineering a prompt engineer. arXiv.10.18653/v1/2024.findings-acl.21 ↗
14. Wang, X., Li, C., Wang, Z., Bai, F., Luo, H., Zhang, J., Jojic, N., Xing, E.P., and Hu, Z. (2023). Promptagent: Strategic planning with language models enables expert-level prompt optimization. arXiv.
15. Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. (2023). Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv.
16. Alonso, O., and Mizzaro, S. (2009, January 23). Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, Boston, MA, USA.
17. Blanco, R., Halpin, H., Herzig, D.M., Mika, P., Pound, J., Thompson, H.S., and Tran Duc, T. (2011, January 24–28). Repeatable and reliable search system evaluation using crowdsourcing. Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Beijing, China.10.1145/2009916.2010039 ↗
18. Maddalena, E., Basaldella, M., De Nart, D., Degl’Innocenti, D., Mizzaro, S., and Demartini, G. (November, January 30). Crowdsourcing relevance assessments: The unexpected benefits of limiting the time to judge. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Austin, TX, USA.
19. Nouri, Z., Wachsmuth, H., and Engels, G. (2020, January 8–13). Mining crowdsourcing problems from discussion forums of workers. Proceedings of the 28th International Conference on Computational Linguistics, Online.10.18653/v1/2020.coling-main.551 ↗
20. Keikha, M., Park, J.H., and Croft, W.B. (2014, January 6–11). Evaluating answer passages using summarization measures. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, Gold Coast, Australia.10.1145/2600428.2609485 ↗
21. Dietz, L., Chatterjee, S., Lennox, C., Kashyapi, S., Oza, P., and Gamari, B. (2022, January 11–15). Wikimarks: Harvesting Relevance Benchmarks from Wikipedia. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain.10.1145/3477495.3531731 ↗
22. Wang, S., Liu, Y., Xu, Y., Zhu, C., and Zeng, M. (2021). Want to reduce labeling cost? GPT-3 can help. arXiv.10.18653/v1/2021.findings-emnlp.354 ↗
23. Ding, B., Qin, C., Liu, L., Bing, L., Joty, S., and Li, B. (2022). Is gpt-3 a good data annotator?. arXiv.10.18653/v1/2023.acl-long.626 ↗
24. MacAvaney, S., and Soldaini, L. (2023). One-Shot Labeling for Automatic Relevance Estimation. arXiv.10.1145/3539618.3592032 ↗
25. Faggioli, G., Dietz, L., Clarke, C., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., and Stein, B. (2023). Perspectives on Large Language Models for Relevance Judgment. arXiv.10.1145/3578337.3605136 ↗
26. Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q.V., Zhou, D., and Chen, X. (2023). Large language models as optimizers. arXiv.
27. Yang, S., Zhao, H., Zhu, S., Zhou, G., Xu, H., Jia, Y., and Zan, H. (2024, January 26–27). Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i17.29907 ↗
28. Ma, R., Wang, X., Zhou, X., Li, J., Du, N., Gui, T., Zhang, Q., and Huang, X. (2024). Are large language models good prompt optimizers?. arXiv.
29. Choi, J. (2024). Identifying Key Terms in Prompts for Relevance Evaluation with GPT Models. arXiv.10.5121/ijnlc.2024.13201 ↗
30. Binary or Graded, Few-Shot or Zero-Shot: Prompt Design for GPTs in Relevance Evaluation
31. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., and Kumar, A. (2022). Holistic evaluation of language models. arXiv.