Can Large-Language Models Replace Humans in Agile Effort Estimation? Lessons from a Controlled Experiment
Source abstract
Abstract
Effort estimation is critical in software engineering to assess the resources needed for development tasks and to enable realistic commitments in agile iterations. This study investigates whether generative AI tools, which are transforming various aspects of software development, can improve effort estimation efficiency. A controlled experiment was conducted in which development teams upgraded an existing information system, with the experimental group using the generative-AI-based tool GitLab Duo for estimation and the control group using conventional methods (e.g., planning poker or analogy-based planning). Results show that while generative-AI-based estimation tools achieved only 16% accuracy—currently insufficient for industry standards—they offered valuable support for task breakdown and iteration planning. Participants noted that a combination of conventional methods and AI-based tools could offer enhanced accuracy and efficiency in future planning.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Engineering Techniques and Practices · Software Engineering Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Electrical Engineering and Computer Science, University of Maribor, Koroška Cesta 46, 2000 Maribor, Slovenia
Funding
Slovenian Research and Innovation Agency
References (32)
Show source-provided references
1. Cambridge University Press (2024, August 09). Cambridge Dictionary. Available online: https://dictionary.cambridge.org.
2. PMI (2013). A Guide to the Project Management Body of Knowledge (PMBOK Guide), Project Management Institute. [5th ed.].
3. Brooks, F. (1995). The Mythical Man-Month: Essays on Software Engineering, Addison-Wesley.
4. An Update on Effort Estimation in Agile Software Development: A Systematic Literature Review10.1109/ACCESS.2020.3021664 ↗
5. Usman, M., Mendes, E., Neiva, F., and Britto, R. (2014, January 17). Effort Estimation in Agile Software Development: A Systematic Literature Review. Proceedings of the 10th International Conference on Predictive Models in Software Engineering, Turin, Italy.10.1145/2639490.2639503 ↗
6. Effort Estimation in Agile Software Development: A Systematic Map Study
7. Schweighofer, T., Kline, A., Pavlič, L., and Heričko, M. (2016, January 29–31). How is Effort Estimated in Agile Software Development Projects?. Proceedings of the SQAMIA, 5th Workshop of Software Quality, Analysis, Monitoring, Improvement, and Applications, Budapest, Hungary.
8. Cohn, M. (2005). Agile Estimating and Planning, Addison-Wesley Professional.
9. Dinwiddie, G. (2019). Software Estimation Without Guessing, Pragmatic Bookshelf.
10. A Review of Agile Software Effort Estimation Methods
11. (2024, August 09). Digital.AI. 15th State of Agile Report. Available online: https://digital.ai/resource-center/analyst-reports/state-of-agile-report.
12. Schwaber, K., and Sutherland, J. (2024, November 03). The Definitive Guide to Scrum: The Rules of the Game. Retrospectiva del Sprint de Nexus. Available online: https://scrum.org.
13. Agile Alliance (2024, November 03). What Is Scrumban?. Available online: https://www.agilealliance.org/scrumban/.
14. (2024, November 03). Large Scale Scrum (LeSS). Available online: https://less.works/.
15. (2024, November 03). Scaled Agile Framework. Available online: https://scaledagileframework.com/.
16. (2024, November 03). Extreme Programming (XP). Available online: http://www.extremeprogramming.org/.
17. (2024, November 03). Feature-Driven Development (FDD). Available online: https://agilemodeling.com/essays/fdd.htm.
18. Data-driven effort estimation techniques of agile user stories: A systematic literature review10.1007/s10462-021-10132-x ↗
19. Beware of botshit: How to manage the epistemic risks of generative chatbots10.1016/j.bushor.2024.03.001 ↗
20. Bhat, M., Haight, C., and Blosen, B. (2024). How Platform Engineering Teams Can Augment DevOps with AI, Gartner Reseaerch.
21. Finnie, G., and Wittig, G. (1996, January 24–27). AI tools for software development effort estimation. Proceedings of the 1996 International Conference Software Engineering: Education and Practice, Dunedin, New Zealand.
22. Reliable machine learning models for estimating effective software development efforts: A comparative analysis10.1016/j.jer.2023.100150 ↗
23. Tran, T.N., Tran, H.T., and Nguyen, Q.N. (2023, January 21–22). Leveraging AI for Enhanced Software Effort Estimation: A Comprehensive Study and Framework Proposal. Proceedings of the 2023 International Conference on the Cognitive Computing and Complex Data (ICCD), Huaian, China.10.1109/ICCD59681.2023.10420603 ↗
24. Koskinen, K. (2021). AI-Assisted Software Development Effort Estimation. [Master’s Thesis, Department of Computing, University of Turku].
25. Hartenstein, S., Leroy Johnson, S., and Schmietendorf, A. (2024). Towards a Fast Cost Estimation Supported by Large Language Models, COSMIC Sizing. Technical Report.
26. Coelho, C.N., Xiong, H., Karayil, T., Koratala, S., Shang, R., Bollinger, J., Shabar, M., and Nair, S. (2024). Effort and Size Estimation in Software Projects with Large Language Model-based Intelligent Interfaces. arXiv.10.22541/au.172115454.45234568/v1 ↗
27. Arman, A., Di Reto, E., Mecella, M., and Santucci, G. (2023, January 14–16). An Approach for Software Development Effort Estimation Using ChatGPT. Proceedings of the 2023 IEEE International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), Paris, France.10.1109/WETICE57085.2023.10477777 ↗
28. Ebrahim, E., Sayed, M., Youssef, M., Essam, H., El-Fattah, S.A., Ashraf, D., Magdy, O., and ElAdawi, R. (2023, January 17–20). AI Decision Assistant ChatBot for Software Release Planning and Optimized Resource Allocation. Proceedings of the 2023 IEEE International Conference on Artificial Intelligence Testing (AITest), Athens, Greece.10.1109/AITest58265.2023.00018 ↗
29. Who is better in project planning?Generative artificial intelligence or project managers?10.1016/j.plas.2023.100101 ↗
30. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., and Wesslén, A. (2012). Experimentation in Software Engineering, Springer.10.1007/978-3-642-29044-2 ↗
31. A practical guide to controlled experiments of software engineering tools with human participants10.1007/s10664-013-9279-3 ↗
32. Better sure than safe? Over-confidence in judgement based software development effort prediction intervals10.1016/S0164-1212(02)00160-7 ↗