A Scoping Review and Assessment Framework for Technical Debt in the Development and Operation of AI/ML Competition Platforms
Source abstract
Abstract
Technical debt (TD) has emerged as a significant concern in the development of AI/ML applications, where rapid experimentation, evolving objectives, and complex data pipelines often introduce hidden quality and maintainability issues. Within this broader context, AI/ML competition platforms face heightened risks due to time-constrained environments and evolving requirements. Despite its relevance, TD in such competitive settings remains underexplored and lacks systematic investigation. This study addresses two research questions: (RQ1) What are the most significant types of technical debt recorded in AI-based systems? and (RQ2) How can we measure the technical debt of an AI-based competition platform? We present a scoping review of 100 peer-reviewed publications related to AI/ML competitions, aiming to map the landscape of TD manifestations and management practices. Through thematic analysis, the study identifies 18 distinct types of technical debt, each accompanied by a definition, rationale, and example grounded in competition scenarios. Based on this typology, a stakeholder-oriented assessment framework is proposed, including a detailed questionnaire and a methodology for the quantitative evaluation of TD across multiple categories. A novel contribution is the introduction of Accessibility Debt, which addresses the challenges associated with the ease and speed of immediate use of the AI/ML competition platforms. The review also incorporates bibliometric insights, revealing the fragmented and uneven treatment of TD across the literature. The findings offer a unified conceptual foundation for future work and provide practical tools for both organizers and participants to systematically detect, interpret, and address technical debt in competitive AI settings, ultimately promoting more sustainable and trustworthy AI research environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Explainable Artificial Intelligence (XAI) · Ethics and Social Impacts of AI · Artificial Intelligence in Healthcare and Education
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Science & Technology, Hellenic Open University, 26335 Patras, Greece
References (100+)
Show source-provided references
1. Software Engineering for AI-Based Systems: A Survey
2. Felderer, M., and Ramler, R. (2021). Quality Assurance for AI-based Systems: Overview and Challenges. arXiv.10.1007/978-3-030-65854-0_3 ↗
3. Sofware engneering challenges for machine learning applications: A literature review
4. Tang, Y., Khatchadourian, R., Bagherzadeh, M., Singh, R., Stewart, A., and Raja, A. (2021, January 25–28). An Empirical Study of Refactorings and Technical Debt in Machine Learning Systems How does access to this work benefit you?. Proceedings of the 021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), Madrid, Spain.10.1109/ICSE43902.2021.00033 ↗
5. Hidden technical debt in machine learning systems
6. Luitse, D.M.R., Blanke, T., and Poell, T. (AoIR Sel. Pap. Internet Res., 2022). Ai Competitions As Infrastructures: Examining Power Relations on Kaggle and Grand Challenge in Ai-Driven Medical Imaging, AoIR Sel. Pap. Internet Res.10.5210/spir.v2022i0.13044 ↗
7. Breck, E., Cai, S., Nielsen, E., Salib, M., and Sculley, D. (2017, January 11–14). The ML test score: A rubric for ML production readiness and technical debt reduction. Proceedings of the 2017 IEEE International Conference on Big Data (Big Data), Boston, MA, USA.10.1109/BigData.2017.8258038 ↗
8. Konen, W. (2019, January 20–23). General board game playing for education and research in generic AI game learning. Proceedings of the 2019 IEEE Conference on Games (CoG), London, UK.10.1109/CIG.2019.8848070 ↗
9. AI World Cup: Robot-Soccer-Based Competitions10.1109/TG.2021.3065410 ↗
10. Attanasio, G., Giobergia, F., Pasini, A., Ventura, F., Baralis, E., Cagliero, L., Garza, P., Apiletti, D., Cerquitelli, T., and Chiusano, S. (2020, January 13–17). DSLE: A Smart Platform for Designing Data Science Competitions. Proceedings of the 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC), Madrid, Spain.10.1109/COMPSAC48688.2020.00026 ↗
11. Zobernig, V., Saldanha, R.A., He, J., van der Sar, E., van Doorn, J., Hua, J.-C., Mason, L.R., Czechowski, A., Indjic, D., and Kosmala, T. (2022). RangL: A Reinforcement Learning Competition Platform. SSRN Electron. J., 1–10.10.2139/ssrn.4168309 ↗
12. Stephenson, M., Piette, E., Soemers, D.J.N.J., and Browne, C. (2019, January 20–23). Ludii as a competition platform. Proceedings of the 2019 IEEE Conference on Games (CoG), London, UK.10.1109/CIG.2019.8848084 ↗
13. Kempka, M., Wydmuch, M., Runc, G., Toczek, J., and Jaskowski, W. (2016, January 20–23). ViZDoom: A Doom-based AI research platform for visual reinforcement learning. Proceedings of the 2016 IEEE Conference on Computational Intelligence and Games (CIG), Santorini, Greece.10.1109/CIG.2016.7860433 ↗
14. Kalles, D. (2016, January 18–20). Artificial intelligence meets software engineering in computing education. Proceedings of the 9th Hellenic Conference on Artificial Intelligence, Thessaloniki, Greece.10.1145/2903220.2903223 ↗
15. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv.
16. How to Run a Successful Game-Based AI Competition10.1109/TCIAIG.2014.2365470 ↗
17. Kästner, C., and Kang, E. (2020, January 5–11). Teaching software engineering for AI-enabled systems. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering Education and Training, Seoul, Republic of Korea.10.1145/3377814.3381714 ↗
18. Chesani, F., Galassi, A., Mello, P., and Trisolini, G. (2017, January 14–17). A game-based competition as instrument for teaching artificial intelligence. Proceedings of the AI* IA 2017 Advances in Artificial Intelligence: XVIth International Conference of the Italian Association for Artificial Intelligence, Bari, Italy.10.1007/978-3-319-70169-1_6 ↗
19. Technical Debt in AI-Enabled Systems: On the Prevalence, Severity, Impact, and Management Strategies for Code and Architecture10.1016/j.jss.2024.112151 ↗
20. Serban, A., and Visser, J. (2022, January 15–18). Adapting Software Architectures to Machine Learning Challenges. Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Honolulu, HI, USA.10.1109/SANER53432.2022.00029 ↗
21. Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., and Zimmermann, T. (2019, January 25–31). Software Engineering for Machine Learning: A Case Study. Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), Montreal, QC, Canada.10.1109/ICSE-SEIP.2019.00042 ↗
22. Sklavenitis, D., and Kalles, D. (2024, January 11–13). Measuring Technical Debt in AI-Based Competition Platforms. Proceedings of the 13th Hellenic Conference on Artificial Intelligence (SETN 2024), Athens, Greece.10.1145/3688671.3688783 ↗
23. Software Engineering of Machine Learning Systems10.1145/3539783 ↗
24. Review of Educational Benefits of Game Jams: Participant and Industry Perspective10.14746/HL.2022.15.7 ↗
25. Game jams for learning and teaching: A review10.4018/IJGBL.2020040104 ↗
26. Principles alone cannot guarantee ethical AI10.1038/s42256-019-0114-4 ↗
27. Foidl, H., Felderer, M., and Ramler, R. (2022). Data Smells: Categories, Causes and Consequences, and Detection of Suspicious Data in AI-Based Systems, Association for Computing Machinery.10.1145/3522664.3528590 ↗
28. Data Validation for Machine Learning
29. Liu, J., Huang, Q., Xia, X., Shihab, E., Lo, D., and Li, S. (2020, January 5–11). Is using deep learning frameworks free?. characterizing technical debt in deep learning frameworks. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Society, Seoul, Republic of Korea.10.1145/3377815.3381377 ↗
30. Bogner, J., Verdecchia, R., and Gerostathopoulos, I. (2021, January 19–21). Characterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Study. Proceedings of the 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), Madrid, Spain.10.1109/TechDebt52882.2021.00016 ↗
31. Software-Engineering Design Patterns for Machine Learning Applications
32. A systematic mapping study on technical debt and its management10.1016/j.jss.2014.12.027 ↗
33. A tertiary study on technical debt: Types, management strategies, research trends, and base information for practitioners10.1016/j.infsof.2018.05.010 ↗
34. Requirements engineering for artificial intelligence systems: A systematic mapping study10.1016/j.infsof.2023.107176 ↗
35. Vogelsang, A., and Borg, M. (2019, January 23–27). Requirements engineering for machine learning: Perspectives from data scientists. Proceedings of the 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW), Jeju, Republic of Korea.10.1109/REW.2019.00050 ↗
36. Warnett, S.J., and Zdun, U. (2022, January 12–15). Architectural Design Decisions for Machine Learning Deployment. Proceedings of the 2022 IEEE 19th International Conference on Software Architecture (ICSA), Honolulu, HI, USA.10.1109/ICSA53651.2022.00017 ↗
37. Heiland, L., Hauser, M., and Bogner, J. (2023). Design Patterns for AI-based Systems: A Multivocal Literature Review and Pattern Repository. arXiv.10.1109/CAIN58948.2023.00035 ↗
38. Washizaki, H., Uchida, H., Khomh, F., and Guéhéneuc, Y.G. (2019, January 13–14). Studying Software Engineering Patterns for Designing Machine Learning Systems. Proceedings of the 2019 10th International Workshop on Empirical Software Engineering in Practice (IWESEP), Tokyo, Japan.10.1109/IWESEP49350.2019.00017 ↗
39. The Five Laws of SE for AI10.1109/MS.2019.2954841 ↗
40. Data lifecycle challenges in production machine learning: A survey10.1145/3299887.3299891 ↗
41. Data collection and quality challenges in deep learning: A data-centric AI perspective10.1007/s00778-022-00775-9 ↗
42. Hutchinson, B., Smart, A., Hanna, A., Denton, E., Greer, C., Kjartansson, O., Barnes, P., and Mitchell, M. (2021, January 3–10). Towards accountability for machine learning datasets: Practices from software engineering and infrastructure. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual.10.1145/3442188.3445918 ↗
43. Data Validation Process in Machine Learning Pipeline
44. Foidl, H., and Felderer, M. (2021, January 3–10). Risk-based data validation in machine learning-based software systems. Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation, Tallinn, Estonia.
45. Survey on AI Sustainability: Emerging Trends on Learning Algorithms and Research Challenges [Review Article]10.1109/MCI.2023.3245733 ↗
46. What Is Really Different in Engineering AI-Enabled Systems?
47. A taxonomy of software engineering challenges for machine learning systems: An empirical investigation
48. Schelter, S., Biessmann, F., Januschowski, T., Salinas, D., Seufert, S., and Szarvas, G. (2018). On Challenges in Machine Learning Model Management. Bull. IEEE Comput. Soc. Tech. Comm. Data Eng., 5–13. Available online: http://sites.computer.org/debull/A18dec/p5.pdf.
49. Towards CRISP-ML (Q): A Machine Learning Process Model with Quality Assurance Methodology10.3390/make3020020 ↗
50. Bailis, P., Olukotun, K., Re, C., and Zaharia, M. (2017). Infrastructure for Usable Machine Learning: The Stanford DAWN Project. arXiv.
51. Machine Learning Testing: Survey, Landscapes and Horizons10.1109/TSE.2019.2962027 ↗
52. Quality Issues in Machine Learning Software Systems10.1007/s10664-024-10536-7 ↗
53. Murphy, C., Kaiser, G., and Arias, M. (2025, June 18). A Framework for Quality Assurance of Machine Learning Applications. pp. 1–10. Available online: https://www.researchgate.net/publication/228687118_A_Framework_for_Quality_Assurance_of_Machine_Learning_Applications.
54. The oracle problem in software testing: A survey10.1109/TSE.2014.2372785 ↗
55. On testing machine learning programs10.1016/j.jss.2020.110542 ↗
56. Golendukhina, V., Lenarduzzi, V., and Felderer, M. (2022, January 16–24). What is Software Quality for AI Engineers?. Towards a Thinning of the Fog. In Proceedings of the 1st International Conference on AI Engineering: Software Engineering for AI, Pittsburgh, Pennsylvania.10.1145/3522664.3528599 ↗
57. Albuquerque, D., Guimaraes, E., Tonin, G., Perkusich, M., Almeida, H., and Perkusich, A. (2022, January 16–18). Comprehending the Use of Intelligent Techniques to Support Technical Debt Management. Proceedings of the International Conference on Technical Debt, Pittsburgh, Pennsylvania.10.1145/3524843.3528097 ↗
58. Njomou, A.T., Fokaefs, M., Silatchom Kamga, D.F., and Adams, B. (2022, January 15–17). On the Challenges of Migrating to Machine Learning Life Cycle Management Platforms. Proceedings of the 32nd Annual International Conference on Computer Science and Software Engineering, Toronto, ON, Canada.
59. Van Der Weide, T., Papadopoulos, D., Smirnov, O., Zielinski, M., and Van Kasteren, T. (2017, January 14–19). Versioning for end-to-end machine learning pipelines. Proceedings of the 1st Workshop on Data Management for End-to-End Machine Learning, Chicago, IL, USA.10.1145/3076246.3076248 ↗
60. Arpteg, A., Brinne, B., Crnkovic-Friis, L., and Bosch, J. (2018, January 29–31). Software engineering challenges of deep learning. Proceedings of the 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Prague, Czech Republic.10.1109/SEAA.2018.00018 ↗
61. Kery, M.B., Radensky, M., Arya, M., John, B.E., and Myers, B.A. (2018, January 21–26). The story in the notebook: Exploratory data science using a literate programming tool. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal QC, Canada.10.1145/3173574.3173748 ↗
62. A software engineering perspective on engineering machine learning systems: State of the art and challenges10.1016/j.jss.2021.111031 ↗
63. OBrien, D., Biswas, S., Imtiaz, S., Abdalkareem, R., Shihab, E., and Rajan, H. (2022). 23 Shades of Self-Admitted Technical Debt: An Empirical Study on Machine Learning Software, Association for Computing Machinery.10.1145/3540250.3549088 ↗
64. Challenges in Deploying Machine Learning: A Survey of Case Studies10.1145/3533378 ↗
65. Van Oort, B., Cruz, L., Aniche, M., and Van Deursen, A. (2021, January 30–31). The prevalence of code smells in machine learning projects. Proceedings of the 2021 IEEE/ACM 1st Workshop on AI Engineering–Software Engineering for AI (WAIN), Madrid, Spain.10.1109/WAIN52551.2021.00011 ↗
66. Gesi, J., Liu, S., Li, J., Ahmed, I., Nagappan, N., Lo, D., de Almeida, E.S., Kochhar, P.S., and Bao, L. (2022). Code Smells in Machine Learning Systems. arXiv.
67. Wang, J., Li, L., and Zeller, A. (2020–19, January 27). Better Code, Better Sharing: On the Need of Analyzing Jupyter Notebooks. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results, Seoul, Republic of Korea.10.1145/3377816.3381724 ↗
68. Pimentel, J.F., Murta, L., Braganholo, V., and Freire, J. (2019, January 25–31). A large-scale study about quality and reproducibility of jupyter notebooks. Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), Montreal, QC, Canada.10.1109/MSR.2019.00077 ↗
69. Haakman, M.P.A. (2025, June 18). Studying the Machine Learning Lifecycle and Improving Code Quality of Machine Learning Applications. Available online: https://repository.tudelft.nl/islandora/object/uuid%3A38ff4e9a-222a-4987-998c-ac9d87880907.
70. De Souza Nascimento, E., Ahmed, I., Oliveira, E., Palheta, M.P., Steinmacher, I., and Conte, T. (2019, January 19–20). Understanding Development Process of Machine Learning Systems: Challenges and Solutions. Proceedings of the2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Porto de Galinhas, Brazil.10.1109/ESEM.2019.8870157 ↗
71. Nahar, N., Zhou, S., Lewis, G., and Kastner, C. (2022, January 21–29). Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process. Proceedings of the 44th International Conference on Software Engineering, Pittsburgh, Pennsylvania.10.1145/3510003.3510209 ↗
72. Exploring the Impact of Code Clones on Deep Learning Software10.1145/3607181 ↗
73. Rios, N., Mendes, L., Cerdeiral, C., Magalhães, A.P.F., Perez, B., Correal, D., Astudillo, H., Seaman, C., Izurieta, C., and Santos, G. (2020). Hearing the Voice of Software Practitioners on Causes, Effects, and Practices to Deal with Documentation Debt. Requirements Engineering: Foundation for Software Quality: 26th International Working Conference, Springer.10.1007/978-3-030-44429-7_4 ↗
74. Chang, J., and Custis, C. (September, January 31). Understanding Implementation Challenges in Machine Learning Documentation. Proceedings of the 2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Gran Canaria, Spain.
75. Shivashankar, K., and Martini, A. (September, January 31). Maintainability Challenges in ML: A Systematic Literature Review. Proceedings of the 2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Gran Canaria, Spain.10.1109/SEAA56994.2022.00018 ↗
76. Tamburri, D.A., Kruchten, P., Lago, P., and Van Vliet, H. (2013, January 25). What is social debt in software engineering?. Proceedings of the 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), San Francisco, CA, USA.10.1109/CHASE.2013.6614739 ↗
77. Mailach, A., and Siegmund, N. (2023, January 14–20). Socio-Technical Anti-Patterns in Building ML-Enabled Software: Insights from Leaders on the Forefront. Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), Melbourne, Australia.10.1109/ICSE48619.2023.00067 ↗
78. Ishikawa, F., and Yoshioka, N. (2019, January 28). How Do Engineers Perceive Difficulties in Engineering of Machine-Learning Systems?—Questionnaire Survey. Proceedings of the 2019 IEEE/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER&IP), Montreal, QC, Canada.10.1109/CESSER-IP.2019.00009 ↗
79. Vakkuri, V., Kemell, K.K., Jantunen, M., and Abrahamsson, P. (2019, January 28). “This is Just a Prototype”: How Ethics Are Ignored in Software Startup-Like Environments. Proceedings of the 2019 IEEE/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER&IP), Montreal, QC, Canada.
80. The Ethics of AI Ethics: An Evaluation of Guidelines10.1007/s11023-020-09517-8 ↗
81. Patterns and antipatterns, principles, and pitfalls: Accountability and transparency in artificial intelligence
82. Who pays for ethical debt in AI?10.1007/s43681-020-00030-3 ↗
83. Bhatia, A., Khomh, F., Adams, B., and Hassan, A.E. (2023). An Empirical Study of Self-Admitted Technical Debt in Machine Learning Software. arXiv.
84. Bavota, G., and Russo, B. (2016, January 14–15). A large-scale empirical study on self-admitted technical debt. Proceedings of the 13th International Conference on Mining Software Repositories, Austin, TX, USA.10.1145/2901739.2901742 ↗
85. Nascimento, E., Nguyen-Duc, A., Sundbø, I., and Conte, T. (2020). Software engineering for artificial intelligence and machine learning software: A systematic literature review. arXiv.
86. How does machine learning change software development practices?
87. Serban, A., Van Der Blom, K., Hoos, H., and Visser, J. (2020, January 5–9). Adoption and effects of software engineering best practices in machine learning. Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Bari, Italy.10.1145/3382494.3410681 ↗
88. SE4AI: A Training Program Considering Technical, Social, and Professional Aspects of AI-Based Software Systems10.1109/MS.2023.3322438 ↗
89. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI10.1016/j.inffus.2019.12.012 ↗
90. Explainable Deep Reinforcement Learning: State of the Art and Challenges
91. Speith, T. (2022, January 21–24). A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea.10.1145/3531146.3534639 ↗
92. Explainable AI (XAI): Core Ideas, Techniques, and Solutions10.1145/3561048 ↗
93. Techniques for interpretable machine learning10.1145/3359786 ↗
94. Puiutta, E., and Veith, E.M.S.P. (2020). Explainable Reinforcement Learning: A Survey. Machine Learning and Knowledge Extraction. CD-MAKE 2020. Lecture Notes in Computer Science, Springer.10.1007/978-3-030-57321-8_5 ↗
95. Mizrahi, M. (2025). AI4Science and the context distinction. AI Ethic, 1–6.10.1007/s43681-025-00677-w ↗
96. Common Pitfalls and Recommendations for Grand Challenges in Medical Artificial Intelligence10.1016/j.euf.2021.05.008 ↗
97. Pavao, A., Guyon, I., Letournel, A.-C., Baró, X., Escalante, H., Escalera, S., Thomas, T., and Xu, Z. (2025, June 18). CodaLab Competitions: An Open Source Platform to Organize Scientific Challenges. Available online: https://hal.inria.fr/hal-03629462/document.
98. Guss, W.H., Castro, M.Y., Devlin, S., Houghton, B., Kuno, N.S., Loomis, C., Milani, S., Mohanty, S., Nakata, K., and Salakhutdinov, R. (2021). The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors. arXiv.
99. Perez-Liebana, D., Samothrakis, S., Togelius, J., Lucas, S.M., and Schaul, T. (March, January 25). General video game AI: Competition, challenges, and opportunities. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA.
100. Kaggle forecasting competitions: An overlooked learning opportunity10.1016/j.ijforecast.2020.07.007 ↗