From DevOps to MLOps: Overview and Application to Electricity Market Forecasting
Source abstract
Abstract
In the Software Development Life Cycle (SDLC), Development and Operations (DevOps) has been proven to deliver reliable, scalable software within a shorter time. Due to the explosion of Machine Learning (ML) applications, the term Machine Learning Operations (MLOps) has gained significant interest among ML practitioners. This paper explains the DevOps and MLOps processes relevant to the implementation of MLOps. The contribution of this paper towards the MLOps framework is threefold: First, we review the state of the art in MLOps by analyzing the related work in MLOps. Second, we present an overview of the leading DevOps principles relevant to MLOps. Third, we derive an MLOps framework from the MLOps theory and apply it to a time-series forecasting application in the hourly day-ahead electricity market. The paper concludes with how MLOps could be generalized and applied to two more use cases with minor changes.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Forecasting Techniques and Applications · Data Stream Mining Techniques · Big Data and Business Intelligence
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical Engineering and Automation, School of Electrical Engineering, Aalto University, 02150 Espoo, Finland
Department of Computer Science, Electrical and Space Engineering, Luleå Tekniska Universitet, 97187 Luleå, Sweden
Funding
Business Finland
References (100+)
Show source-provided references
1. Pang, C., Hindle, A., and Barbosa, D. (2020, January 5–11). Understanding DevOps education with Grounded theory. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings, Seul, Korea.10.1145/3377812.3390808 ↗
2. Macarthy, R.W., and Bass, J.M. (2020, January 26–28). An Empirical Taxonomy of DevOps in Practice. Proceedings of the 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Portorož, Slovenia.10.1109/SEAA51224.2020.00046 ↗
3. Wahaballa, A., Wahballa, O., Abdellatief, M., Xiong, H., and Qin, Z. (2015, January 23–25). Toward unified DevOps model. Proceedings of the 2015 6th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China.10.1109/ICSESS.2015.7339039 ↗
4. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.F., and Dennison, D. (2015, January 7–12). Hidden technical debt in machine learning systems. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
5. Continuous software engineering: A roadmap and agenda10.1016/j.jss.2015.06.063 ↗
6. Haindl, P., and Plosch, R. (2019, January 25–29). Towards Continuous Quality: Measuring and Evaluating Feature-Dependent Non-Functional Requirements in DevOps. Proceedings of the 2019 IEEE International Conference on Software Architecture Companion (ICSA-C), Hamburg, Germany.10.1109/ICSA-C.2019.00024 ↗
7. Steffens, A., Lichter, H., and Döring, J.S. (2018, January 29). Designing a next-generation continuous software delivery system. Proceedings of the 4th International Workshop on Rapid Continuous Software Engineering, Gothenburg, Sweden.10.1145/3194760.3194768 ↗
8. Building A Platform for Machine Learning Operations from Open Source Frameworks10.1016/j.ifacol.2021.04.161 ↗
9. Makinen, S., Skogstrom, H., Laaksonen, E., and Mikkonen, T. (2021, January 30–31). Who needs MLOps: What data scientists seek to accomplish and how can MLOps help?. Proceedings of the 2021 IEEE/ACM 1st Workshop on AI Engineering—Software Engineering for AI, WAIN, Madrid, Spain.10.1109/WAIN52551.2021.00024 ↗
10. Khan, M.A., Parveen, A., and Sadiq, M. (2014, January 7–8). A method for the selection of software development life cycle models using analytic hierarchy process. Proceedings of the 2014 International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT), Ghaziabad, India.10.1109/ICICICT.2014.6781338 ↗
11. Ben-Zahia, M.A., and Jaluta, I. (2014, January 14-16). Criteria for selecting software development models. Proceedings of the 2014 Global Summit on Computer & Information Technology (GSCIT), Sousse, Tunisia.10.1109/GSCIT.2014.6970099 ↗
12. Selection of appropriate software development life cycle using fuzzy logic10.3233/IFS-120686 ↗
13. Lekh, R. (2015, January 19–20). Pooja Exhaustive study of SDLC phases and their best praxctices to create CDP model for process improvement. Proceedings of the 2015 International Conference on Advances in Computer Engineering and Applications, Ghaziabad, India.10.1109/ICACEA.2015.7164852 ↗
14. Dayal Chauhan, B., Rana, A., and Sharma, N.K. (2017, January 20–22). Impact of development methodology on cost & risk for development projects. Proceedings of the 2017 6th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India.10.1109/ICRITO.2017.8342436 ↗
15. Improving the Quality of Software Development Process by Introducing a New Methodology–AZ-Model10.1109/ACCESS.2017.2787981 ↗
16. Statistical Analysis of the Effects of Heavyweight and Lightweight Methodologies on the Six-Pointed Star Model10.1109/ACCESS.2018.2805702 ↗
17. A Hybrid Software Development Method For Large-Scale Projects: Rational Unified Process With Scrum
18. Velmourougan, S., Dhavachelvan, P., Baskaran, R., and Ravikumar, B. (2014, January 24–27). Software development Life cycle model to build software applications with usability. Proceedings of the 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Delhi, India.10.1109/ICACCI.2014.6968610 ↗
19. Fisher, K.G., and Bankston, A. (2009, January 24–28). From Cradle to Sprint: Creating a Full-Lifecycle Request Pipeline at Nationwide Insurance. Proceedings of the 2009 Agile Conference, Chicago, IL, USA.10.1109/AGILE.2009.72 ↗
20. Driving Agile Architecting with Cost and Risk10.1109/MS.2014.111 ↗
21. Owais, M., and Ramakishore, R. (2016, January 11–13). Effort, duration and cost estimation in agile software development. Proceedings of the 2016 Ninth International Conference on Contemporary Computing (IC3), Noida, India.10.1109/IC3.2016.7880216 ↗
22. Webster, C., Shi, N., and Smith, I.S. (2012, January 3–10). Delivering software into NASA’s Mission Control Center using agile development techniques. Proceedings of the 2012 IEEE Aerospace Conference, Big Sky, MT, USA.10.1109/AERO.2012.6187329 ↗
23. Safety Stories in Agile Development10.1109/MS.2017.108 ↗
24. Alyahya, S., Bin-Hezam, R., and Maddeh, M. (2022). Supporting Remote Customer Involvement in Distributed Agile Development: A Coordination Approach. IEEE Trans. Eng. Manag., 1–14.
25. Kumar, G., and Bhatia, P.K. (2014, January 8–9). Comparative Analysis of Software Engineering Models from Traditional to Modern Methodologies. Proceedings of the 2014 Fourth International Conference on Advanced Computing & Communication Technologies, Rohtak, India.10.1109/ACCT.2014.73 ↗
26. Sinha, A., and Das, P. (2018, January 4–5). Agile Methodology Vs. Traditional Waterfall SDLC: A case study on Quality Assurance process in Software Industry. Proceedings of the 2021 5th International Conference on Electronics, Materials Engineering & Nano-Technology (IEMENTech), Kolkata, India.10.1109/IEMENTech53263.2021.9614779 ↗
27. Ahmed, A., Ahmad, S., Ehsan, N., Mirza, E., and Sarwar, S.Z. (2010, January 2–5). Agile software development: Impact on productivity and quality. Proceedings of the 2010 IEEE International Conference on Management of Innovation & Technology, Singapore.10.1109/ICMIT.2010.5492703 ↗
28. Tommy, R., Mhaisekar, M., Kallepally, S., Varghese, L., Ahmed, S., and Somaraju, M.D. (2015, January 2–3). Dynamic quality control in agile methodology for improving the quality. Proceedings of the 2015 IEEE International Conference on Computer Graphics, Vision and Information Security (CGVIS), Bhubaneshwar, India.10.1109/CGVIS.2015.7449927 ↗
29. Darrin, M.A.G., and Devereux, W.S. (2017, January 24–27). The Agile Manifesto, design thinking and systems engineering. Proceedings of the 2017 Annual IEEE International Systems Conference (SysCon), Montreal, QC, Canada.10.1109/SYSCON.2017.7934765 ↗
30. Get ready for agile methods, with care10.1109/2.976920 ↗
31. Empirical studies of agile software development: A systematic review10.1016/j.infsof.2008.01.006 ↗
32. Samarawickrama, S.S., and Perera, I. (2017, January 6–9). Continuous scrum: A framework to enhance scrum with DevOps. Proceedings of the 2017 Seventeenth International Conference on Advances in ICT for Emerging Regions (ICTer), Colombo, Sri Lanka.10.1109/ICTER.2017.8257808 ↗
33. Miller, A. (2008, January 4–8). A hundred days of continuous integration. Proceedings of the Agile 2008 Conference, Toronto, ON, Canada.10.1109/Agile.2008.8 ↗
34. Arachchi, S.A.I.B.S., and Perera, I. (2020, January 28–30). Continuous Integration and Continuous Delivery Pipeline Automation for Agile Software Project Management. Proceedings of the 2018 Moratuwa Engineering Research Conference (MERCon), Moratuwa, Sri Lanka.10.1109/MERCon.2018.8421965 ↗
35. Süß, J.G., Swift, S., and Escott, E. (2022). Using DevOps toolchains in Agile model-driven engineering. Softw. Syst. Model., 1–16.10.1007/s10270-022-01003-2 ↗
36. Nagarajan, A.D., and Overbeek, S.J. (2018). A DevOps Implementation Framework for Large Agile-Based Financial Organizations. OTM Confederated International Conferences On the Move to Meaningful Internet Systems, Springer International Publishing.10.1007/978-3-030-02610-3_10 ↗
37. From Agile to DevOps: Smart Skills and Collaborations10.1007/s10796-019-09905-1 ↗
38. Marrero, L., and Astudillo, H. (2021, January 15–19). DevOps-RAF: An assessment framework to measure DevOps readiness in software organizations. Proceedings of the 2021 40th International Conference of the Chilean Computer Science Society (SCCC), La Serena, Chile.10.1109/SCCC54552.2021.9650363 ↗
39. Pan, W., and Wei, H. (2012, January 12–14). Research on Key Performance Indicator (KPI) of Business Process. Proceedings of the 2012 Second International Conference on Business Computing and Global Informatization, Shanghai, China.10.1109/BCGIN.2012.46 ↗
40. Durga Prasad, N.V.P.R., and Radhakrishna, C. (2019, January 27–28). New Key Performance Indicators (KPI) For Substation Maintenance Performance. Proceedings of the 2019 International Conference on Computing, Power and Communication Technologies (GUCON), New Delhi, India.
41. Zhu, H., and Bayley, I. (2018, January 26–29). If Docker is the Answer, What is the Question?. Proceedings of the 2018 IEEE Symposium on Service-Oriented System Engineering (SOSE), Bamberg, Germany.10.1109/SOSE.2018.00027 ↗
42. Weyuker, E.J., and Avritzer, A. (2002, January 4–7). A metric to predict software scalability. Proceedings of the Eighth IEEE Symposium on Software Metrics, Ottawa, ON, Canada.
43. Dearle, A. (2007, January 23–25). Software Deployment, Past, Present and Future. Proceedings of the Future of Software Engineering (FOSE ’07), Minneapolis, MN, USA.10.1109/FOSE.2007.20 ↗
44. Shahin, M., Babar, M.A., Zahedi, M., and Zhu, L. (2018, January 11–12). Beyond Continuous Delivery: An Empirical Investigation of Continuous Deployment Challenges. Proceedings of the 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Oulu, Finland.10.1109/ESEM.2017.18 ↗
45. Unger-Windeler, C., Klunder, J., and Schneider, K. (2019, January 25). A Mapping Study on Product Owners in Industry: Identifying Future Research Directions. Proceedings of the 2019 IEEE/ACM International Conference on Software and System Processes (ICSSP), Montreal, QC, Canada.10.1109/ICSSP.2019.00026 ↗
46. Tsunoda, M., Matsumura, T., and Matsumoto, K. (2010, January 18–20). Modeling Software Project Monitoring with Stakeholders. Proceedings of the 2010 IEEE/ACIS 9th International Conference on Computer and Information Science, Kaminoyama, Japan.10.1109/ICIS.2010.84 ↗
47. Riungu-Kalliosaari, L., Mäkinen, S., Lwakatare, L.E., Tiihonen, J., and Männistö, T. (2016). DevOps adoption benefits and challenges in practice: A case study. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer.10.1007/978-3-319-49094-6_44 ↗
48. Problems in the interplay of development and IT operations in system development projects: A Delphi study of Norwegian IT experts10.1016/j.infsof.2010.12.002 ↗
49. Tessem, B., and Iden, J. (2008, January 13). Cooperation between developers and operations in software engineering projects. Proceedings of the 2008 International Workshop on Cooperative and Human Aspects of Software Engineering, Leipzig, Germany.10.1145/1370114.1370141 ↗
50. Aligning Architecture Work with Agile Teams10.1109/MS.2015.119 ↗
51. Govil, N., Saurakhia, M., Agnihotri, P., Shukla, S., and Agarwal, S. (2020, January 15–17). Analyzing the Behaviour of Applying Agile Methodologies & DevOps Culture in e-Commerce Web Application. Proceedings of the 2020 4th International Conference on Trends in Electronics and Informatics (ICOEI) (48184), Tirunelveli, India.10.1109/ICOEI48184.2020.9142895 ↗
52. Khmelevsky, Y., Li, X., and Madnick, S. (2017, January 24–27). Software development using agile and scrum in distributed teams. Proceedings of the 2017 Annual IEEE International Systems Conference (SysCon), Montreal, QC, Canada.10.1109/SYSCON.2017.7934766 ↗
53. DevOps10.1109/MS.2016.68 ↗
54. Agrawal, P., and Rawat, N. (2019, January 27–28). Devops, A New Approach To Cloud Development & Testing. Proceedings of the 2019 International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT), Ghaziabad, India.10.1109/ICICT46931.2019.8977662 ↗
55. Choudhary, B., and Rakesh, S.K. (2016, January 3–5). An approach using agile method for software development. Proceedings of the 2016 International Conference on Innovation and Challenges in Cyber Security (ICICCS-INBUSH), Greater Noida, India.10.1109/ICICCS.2016.7542304 ↗
56. What Do We Know about Agile Software Development?10.1109/MS.2009.145 ↗
57. Jabbari, R., Bin Ali, N., Petersen, K., and Tanveer, B. (2016, January 24). What is DevOps?. Proceedings of the Scientific Workshop Proceedings of XP2016, Edinburgh, UK.10.1145/2962695.2962707 ↗
58. Gokarna, M., and Singh, R. (2021, January 19–20). DevOps: A Historical Review and Future Works. Proceedings of the 2021 International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), Greater Noida, India.10.1109/ICCCIS51004.2021.9397235 ↗
59. Sharma, S., Kumar, D., and Fayad, M.E. (2020, January 4–5). An Impact Assessment of Agile Ceremonies on Sprint Velocity Under Agile Software Development. Proceedings of the 2021 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India.10.1109/ICRITO51393.2021.9596508 ↗
60. Srivastava, A., Bhardwaj, S., and Saraswat, S. (2017, January 5–6). SCRUM model for agile methodology. Proceedings of the 2017 International Conference on Computing, Communication and Automation (ICCCA), Greater Noida, India.10.1109/CCAA.2017.8229928 ↗
61. Paasivaara, M., Durasiewicz, S., and Lassenius, C. (2008, January 17–20). Distributed Agile Development: Using Scrum in a Large Project. Proceedings of the 2008 IEEE International Conference on Global Software Engineering, Bangalore, India.10.1109/ICGSE.2008.38 ↗
62. John, M.M., Olsson, H.H., and Bosch, J. (2021, January 1–3). Towards MLOps: A Framework and Maturity Model. Proceedings of the 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Palermo, Italy.10.1109/SEAA53835.2021.00050 ↗
63. Kang, H., Le, M., and Tao, S. (2016, January 4–8). Container and microservice driven design for cloud infrastructure DevOps. Proceedings of the 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, Berlin, Germany.10.1109/IC2E.2016.26 ↗
64. (2022, August 26). Kubeflow–ML Toolkit. Available online: https://www.kubeflow.org/.
65. Zhou, Y., Yu, Y., and Ding, B. (2020, January 23–25). Towards MLOps: A Case Study of ML Pipeline Platform. Proceedings of the 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE), Beijing, China.10.1109/ICAICE51518.2020.00102 ↗
66. Karamitsos, I., Albarhami, S., and Apostolopoulos, C. (2020). Applying DevOps Practices of Continuous Automation for Machine Learning. Information, 11.10.3390/info11070363 ↗
67. Banerjee, A., Chen, C.C., Hung, C.C., Huang, X., Wang, Y., and Chevesaran, R. (August, January 28). Challenges and experiences with MLOps for performance diagnostics in hybrid-cloud enterprise software deployments. Proceedings of the OpML 2020—2020 USENIX Conference on Operational Machine Learning, Online.
68. Tamburri, D.A. (2020, January 1–4). Sustainable MLOps: Trends and Challenges. Proceedings of the 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, SYNASC, Timisoara, Romania.10.1109/SYNASC51798.2020.00015 ↗
69. Raj, E., Buffoni, D., Westerlund, M., and Ahola, K. (2021, January 4–8). Edge MLOps: An Automation Framework for AIoT Applications. Proceedings of the 2021 IEEE International Conference on Cloud Engineering (IC2E), San Francisco, CA, USA.10.1109/IC2E52221.2021.00034 ↗
70. Granlund, T., Kopponen, A., Stirbu, V., Myllyaho, L., and Mikkonen, T. (2021, January 22–30). MLOps challenges in multi-organization setup: Experiences from two real-world cases. Proceedings of the 2021 IEEE/ACM 1st Workshop on AI Engineering—Software Engineering for AI, WAIN 2021, Madrid, Spain.10.1109/WAIN52551.2021.00019 ↗
71. (2022, August 26). Azure MLOps. Available online: https://azure.microsoft.com/en-us/services/machine-learning/mlops/.
72. (2022, August 26). AWS MLOps—Sagemaker. Available online: https://aws.amazon.com/sagemaker/mlops/.
73. (2022, August 26). Practitioners Guide to MLOps—Google Cloud. Available online: https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf.
74. Dyck, A., Penners, R., and Lichter, H. (2015, January 19). Towards Definitions for Release Engineering and DevOps. Proceedings of the 2015 IEEE/ACM 3rd International Workshop on Release Engineering, Florence, Italy.10.1109/RELENG.2015.10 ↗
75. (2022, August 26). What is Devops. Available online: https://www.atlassian.com/devops.
76. Cois, C.A., Yankel, J., and Connell, A. (2014, January 13–15). Modern DevOps: Optimizing software development through effective system interactions. Proceedings of the 2014 IEEE International Professional Communication Conference (IPCC), Pittsburgh, PA, USA.10.1109/IPCC.2014.7020388 ↗
77. Virmani, M. (2015, January 20–22). Understanding DevOps & bridging the gap from continuous integration to continuous delivery. Proceedings of the Fifth International Conference on the Innovative Computing Technology (INTECH 2015), Galicia, Spain.10.1109/INTECH.2015.7173368 ↗
78. Kerzazi, N., Adams MCIS, B., and Montreal, P. (2016, January 14–15). Who Needs Release and DevOps Engineers, and Why?. Proceedings of the International Workshop on Continuous Software Evolution and Delivery, Austin, TX, USA.10.1145/2896941.2896957 ↗
79. Bankar, S., and Shah, D. (2021, January 15–16). Blockchain based framework for Software Development using DevOps. Proceedings of the 2021 4th Biennial International Conference on Nascent Technologies in Engineering (ICNTE), Navi Mumbai, India.10.1109/ICNTE51185.2021.9487760 ↗
80. Soni, M. (2015, January 25–27). End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery. Proceedings of the 2015 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM), Bangalore, India.10.1109/CCEM.2015.29 ↗
81. Quality-Aware DevOps Research: Where Do We Stand?10.1109/ACCESS.2021.3064867 ↗
82. Dalpiaz, F., and Brinkkemper, S. (2018, January 20–24). Agile Requirements Engineering with User Stories. Proceedings of the 2018 IEEE 26th International Requirements Engineering Conference (RE), Banff, AL, Canada.10.1109/RE.2018.00075 ↗
83. Sarkan, H.M., Ahmad, T.P.S., and Bakar, A.A. (2011, January 27–29). Using JIRA and Redmine in requirement development for agile methodology. Proceedings of the 2011 Malaysian Conference in Software Engineering, Kuantan, Malaysia.10.1109/MySEC.2011.6140707 ↗
84. Perera, P., Silva, R., and Perera, I. (2017, January 6–9). Improve software quality through practicing DevOps. Proceedings of the 2017 Seventeenth International Conference on Advances in ICT for Emerging Regions (ICTer), Colombo, Sri Lanka.10.1109/ICTER.2017.8257807 ↗
85. Majumdar, R., Jain, R., Barthwal, S., and Choudhary, C. (2017, January 20–22). Source code management using version control system. Proceedings of the 2017 6th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India.10.1109/ICRITO.2017.8342438 ↗
86. Ren, Y., Xing, T., Quan, Q., and Zhao, Y. (2010, January 26–28). Software Configuration Management of Version Control Study Based on Baseline. Proceedings of the 2010 3rd International Conference on Information Management, Innovation Management and Industrial Engineering, Kunming, China.10.1109/ICIII.2010.506 ↗
87. Shah, J., Dubaria, D., and Widhalm, J. (2018, January 8–10). A Survey of DevOps tools for Networking. Proceedings of the 2018 9th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), New York, NY, USA.10.1109/UEMCON.2018.8796814 ↗
88. Git10.1109/MS.2012.61 ↗
89. Essential Tools: Version Control Systems10.1109/MCSE.2009.194 ↗
90. Paez, N. (2018, January 28–30). Versioning Strategy for DevOps Implementations. Proceedings of the 2018 Congreso Argentino de Ciencias de la Informática y Desarrollos de Investigación (CACIDI), Buenos Aires, Argentina.10.1109/CACIDI.2018.8584362 ↗
91. Palihawadana, S., Wijeweera, C.H., Sanjitha, M.G.T.N., Liyanage, V.K., Perera, I., and Meedeniya, D.A. (2017, January 29–31). Tool support for traceability management of software artefacts with DevOps practices. Proceedings of the 2017 Moratuwa Engineering Research Conference (MERCon), Moratuwa, Sri Lanka.10.1109/MERCon.2017.7980469 ↗
92. (2022, August 26). Git branching Model. Available online: https://nvie.com/posts/a-successful-git-branching-model/.
93. Chang, C.-Y., Ou, P.-P., and Deng, D.-J. (2019, January 18–20). Cross-Site Large-Scale Software Delivery with Enhanced Git Branch Model. Proceedings of the 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS), Beijing, China.10.1109/ICSESS47205.2019.9040834 ↗
94. Version control10.1109/MS.2006.32 ↗
95. (2022, August 26). Branching vs Trunk Based Development. Available online: https://launchdarkly.com/blog/git-branching-strategies-vs-trunk-based-development/.
96. Package Management Systems10.1109/MS.2012.38 ↗
97. Lessons from building static analysis tools at Google10.1145/3188720 ↗
98. Jebbar, O., Saied, M.A., Khendek, F., and Toeroe, M. (2019, January 22–27). Poster: Re-Testing Configured Instances in the Production Environment—A Method for Reducing the Test Suite. Proceedings of the 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), Xi’an, China.10.1109/ICST.2019.00044 ↗
99. Chen, W., Ye, K., Wang, Y., Xu, G., and Xu, C.-Z. (2018, January 11–13). How Does the Workload Look Like in Production Cloud? Analysis and Clustering of Workloads on Alibaba Cluster Trace. Proceedings of the 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS), Singapore.10.1109/PADSW.2018.8644579 ↗
100. Neitsch, A., Wong, K., and Godfrey, M.W. (2012, January 23–28). Build system issues in multilanguage software. Proceedings of the 2012 28th IEEE International Conference on Software Maintenance (ICSM), Trento, Italy.10.1109/ICSM.2012.6405265 ↗