AgentReport: A Multi-Agent LLM Approach for Automated and Reproducible Bug Report Generation
Source abstract
Abstract
Bug reports in open-source projects are often incomplete or low in quality, which reduces maintenance efficiency. To address this issue, we propose AgentReport, a multi-agent pipeline based on large language models (LLMs). AgentReport integrates QLoRA-4bit lightweight fine-tuning, CTQRS (Completeness, Traceability, Quantifiability, Reproducibility, Specificity) structured prompting, Chain-of-Thought reasoning, and one-shot exemplar within seven modules: Data, Prompt, Fine-tuning, Generation, Evaluation, Reporting, and Controller. Using 3966 summary–report pairs from Bugzilla, AgentReport achieved 80.5% in CTQRS, 84.6% in ROUGE-1 Recall, 56.8% in ROUGE-1 F1, and 86.4% in Sentence-BERT (SBERT). Compared with the baseline (77.0% CTQRS, 61.0% ROUGE-1 Recall, 85.0% SBERT), AgentReport improved CTQRS by 3.5 percentage points, Recall by 23.6 points, and SBERT by 1.4 points. The inclusion of F1 complemented Recall-only evaluation, offering a balanced framework that covers structural completeness (CTQRS), lexical coverage and precision (ROUGE-1 Recall/F1), and semantic consistency (SBERT). This modular design enables consistent experimentation and flexible scaling, providing practical evidence that multi-agent LLM pipelines can generate higher-quality bug reports for software maintenance.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Engineering Research · Software Engineering Techniques and Practices · Scientific Computing and Data Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Applied Mathematics, Hankyong National University, Anseong 17579, Republic of Korea
Department of Computer Applied Mathematics, Computer System Institute, Hankyong National University, Anseong 17579, Republic of Korea
References (56)
Show source-provided references
1. Acharya, A., and Ginde, R. (May, January 27). RAG-based bug report generation with large language models. Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), Ottawa, ON, Canada.
2. Bettenburg, N., Just, S., Schröter, A., Weiss, C., Premraj, R., and Zimmermann, T. (2008, January 9–14). What makes a good bug report?. Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE-16), Atlanta, GA, USA.10.1145/1453101.1453146 ↗
3. Medeiros, M., Kulesza, U., Coelho, R., Bonifacio, R., Treude, C., and Barbosa, E.A. (2024). The impact of bug localization based on crash report mining: A developers’ perspective. arXiv.10.1145/3639477.3639730 ↗
4. Chaff from the wheat: Characterizing and determining valid bug reports10.1109/TSE.2018.2864217 ↗
5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020, January 6–12). Language models are few-shot learners. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual.
6. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., and Altman, S. (2023). GPT-4 technical report. arXiv.
7. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., and Lundberg, S. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv.
8. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-rank adaptation of large language models. arXiv.
9. Zhang, H., Zhao, Y., Yu, S., and Chen, Z. (2022, January 4–5). Automated quality assessment for crowdsourced test reports based on dependency parsing. Proceedings of the 9th International Conference on Dependable Systems and Their Applications (DSA), Wulumuqi, China.10.1109/DSA56465.2022.00014 ↗
10. Lin, C.-Y. (2004, January 25–26). ROUGE: A package for automatic evaluation of summaries. Proceedings of the Workshop on Text Summarization Branches Out (ACL), Barcelona, Spain.
11. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. arXiv.
12. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., and Zhou, D. (December, January 28). Chain-of-thought prompting elicits reasoning in large language models. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.
13. Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., and Lin, Y. (2024). A survey on large language model based autonomous agents. Front. Comput. Sci., 18.10.1007/s11704-024-40231-1 ↗
14. Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., and Wen, J. (2024). A survey on the memory mechanism of LLM-based agents. arXiv.
15. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R., and Cao, Y. (2022). ReAct: Synergizing reasoning and acting in language models. arXiv.
16. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., and Liu, J. (2023). AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv.
17. Pandya, K., and Holia, M. (2023). Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizations. arXiv.
18. Reimers, N., and Gurevych, I. (2019, January 3–7). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. Proceedings of the Empirical Methods in Natural Language Processing (EMNLP), Hong Kong, China.10.18653/v1/D19-1410 ↗
19. Sun, C., Lo, D., and Khoo, X. (2011, January 6–12). Towards more accurate retrieval of duplicate bug reports. Proceedings of the ACM International Conference on Systems and Software Engineering (ASE), Lawrence, KS, USA.10.1109/ASE.2011.6100061 ↗
20. A contextual approach towards more accurate duplicate bug report detection and ranking
21. He, Z., Marcus, A., and Poshyvanyk, D. (July, January 30). Using information retrieval and NLP to classify bug reports. Proceedings of the International Conference on Program Comprehension (ICPC), Braga, Portugal.
22. Cubranic, D., and Murphy, G.C. (2004, January 20–24). Automatic bug triage using text categorization. Proceedings of the Sixteenth International Conference on Software Engineering & Knowledge Engineering (SEKE), Banff, AB, Canada.
23. Rastkar, S., Murphy, G.C., and Murray, G. (2010, January 2–8). Summarizing software artifacts: A case study of bug reports. Proceedings of the 32nd International Conference on Software Engineering (ICSE), Cape Town, South Africa.10.1145/1806799.1806872 ↗
24. Mani, S., Catherine, R., Sinha, V.S., and Dubey, A. (2012, January 10–17). AUSUM: Approach for unsupervised bug report summarization. Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (FSE), Cary, NC, USA.10.1145/2393596.2393607 ↗
25. Automatic summarization of bug reports10.1109/TSE.2013.2297712 ↗
26. Modeling the ‘hurried’ bug report reading process to summarize bug reports10.1007/s10664-014-9311-2 ↗
27. GitHub (2025, September 21). Bug Report Summarization Benchmark Dataset. Available online: https://github.com/GindeLab/Ease_2025_AI_model.
28. Beijing Academy of Artificial Intelligence (BAAI) (2023). BAAI BAAI/bge-large-en-v1.5 (English) Embedding Model, Beijing Academy of Artificial Intelligence (BAAI). Available online: https://huggingface.co/BAAI/bge-large-en-v1.5.
29. Team, Q. (2024). Qwen2.5 technical report. arXiv.
30. Team, U. (2025, September 21). Unsloth: Efficient Fine-Tuning Framework for LLMs. Available online: https://github.com/unslothai/unsloth.
31. Billion-scale similarity search with GPUs10.1109/TBDATA.2019.2921572 ↗
32. Effective prediction of bug-fixing priority via weighted graph convolutional networks
33. Fang, S., Zhang, T., Tan, Y., Jiang, H., Xia, X., and Sun, X. (2023, January 14–20). RepresentThemAll: A universal learning representation of bug reports. Proceedings of the IEEE/ACM 45th International Conference on Software Engineering (ICSE), Melbourne, Australia.10.1109/ICSE48619.2023.00060 ↗
34. Liu, H., Yu, Y., Li, S., Guo, Y., Wang, D., and Mao, X. (2020, January 13–15). BugSum: Deep context understanding for bug report summarization. Proceedings of the IEEE/ACM International Conference on Program Comprehension (ICPC), Seoul, Republic of Korea.10.1145/3387904.3389272 ↗
35. Towards effective bug report summarization by domain-specific representation learning10.1109/ACCESS.2024.3368915 ↗
36. Lamkanfi, A., Demeyer, S., Giger, E., and Goethals, B. (2010, January 2–3). Predicting the severity of a reported bug. Proceedings of the IEEE International Conference on Mining Software Repositories (MSR), Cape Town, South Africa.10.1109/MSR.2010.5463284 ↗
37. Sarkar, A., Rigby, P.C., and Bartalos, B. (October, January 30). Improving Bug Triaging with High Confidence Predictions at Ericsson. Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME), Cleveland, OH, USA.
38. Chen, M. (2021). Evaluating large language models trained on code. arXiv.
39. Allal, L.B., Muennighoff, N., Umapathi, L.K., Lipkin, B., and von Werra, L. (2023). StarCoder: Open source code LLMs. arXiv.
40. Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y., Liu, J., Sauvestre, R., and Remez, T. (2023). Code LLaMA: Open foundation models for code. arXiv.
41. Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. (2023). WizardCoder: Empowering code LLMs to speak code fluently. arXiv.
42. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., and Yang, Y. (2023). Self-Refine: Iterative refinement with feedback. arXiv.
43. Chen, X., Lin, M., Schärli, N., and Zhou, D. (2023). Teaching LLMs to self-debug. arXiv.
44. Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., and Chen, W. (2023). CRITIC: LLMs can self-correct with tool-interactive critiquing. arXiv.
45. Nye, M., Andreassen, A.J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., and Luan, D. (2021). Show your work: Scratchpads for intermediate reasoning. arXiv.
46. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002, January 6–7). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
47. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2020, January 26–30). BERTScore: Evaluating text generation with BERT. Proceedings of the 8th International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia.
48. SummEval: Re-evaluating summarization evaluation10.1162/tacl_a_00373 ↗
49. Eghbali, A., and Pradel, M. (2022, January 10–12). CrystalBLEU: Precisely and efficiently measuring code generation quality. Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, Belfast, UK.10.1145/3551349.3556903 ↗
50. Zhou, S., Alon, U., Agarwal, S., and Neubig, G. (2023). CodeBERTScore: Evaluating code generation with BERT-based similarity. arXiv.10.18653/v1/2023.emnlp-main.859 ↗
51. Fu, J., Ng, S.-K., Jiang, Z., and Liu, P. (2023). GPTScore: Evaluate as you desire. arXiv.10.18653/v1/2024.naacl-long.365 ↗
52. Wang, Z., Zhou, S., Fried, D., and Neubig, G. (2022). Execution-based evaluation for open-domain code generation. arXiv.10.18653/v1/2023.findings-emnlp.89 ↗
53. Lu, S., Guo, D., Ren, R., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., and Tang, D. (2021). CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. arXiv.
54. Yasunaga, M., and Liang, P. (2021, January 18–24). Break-it-fix-it: Unsupervised learning for program repair. Proceedings of the 38th International Conference on Machine Learning (ICML), Virtual.
55. A comprehensive study of automated program repair on QuixBugs10.1016/j.jss.2020.110825 ↗
56. GitHub (2025, November 01). AgentReport: A Multi-Agent LLM Approach for Automated and Reproducible Bug Report Generation. Available online: https://github.com/insui12/AgentReport.