Demystifying MLOps and Presenting a Recipe for the Selection of Open-Source Tools
Source abstract
Abstract
Nowadays, machine learning projects have become more and more relevant to various real-world use cases. The success of complex Neural Network models depends upon many factors, as the requirement for structured and machine learning-centric project development management arises. Due to the multitude of tools available for different operational phases, responsibilities and requirements become more and more unclear. In this work, Machine Learning Operations (MLOps) technologies and tools for every part of the overall project pipeline, as well as involved roles, are examined and clearly defined. With the focus on the inter-connectivity of specific tools and comparison by well-selected requirements of MLOps, model performance, input data, and system quality metrics are briefly discussed. By identifying aspects of machine learning, which can be reused from project to project, open-source tools which help in specific parts of the pipeline, and possible combinations, an overview of support in MLOps is given. Deep learning has revolutionized the field of Image processing, and building an automated machine learning workflow for object detection is of great interest for many organizations. For this, a simple MLOps workflow for object detection with images is portrayed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Data Stream Mining Techniques · Anomaly Detection Techniques and Applications · Image and Video Quality Assessment
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute for Data Science, Cloud Computing and IT-Security (IDACUS), Hochschule Furtwangen University, 78120 Furtwangen im Schwarzwald, Germany
Institut de Recherche en Informatique, Mathématiques, Automatique et Signal (IRIMAS), Université de Haute-Alsace, 61 Rue Albert Camus, 68093 Mulhouse, France
References (89)
Show source-provided references
1. Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems, Curran Associates, Inc.
2. Goyal, A. (2020). Machine Learning Operations. International Journal of Information Technology Insights & Transformations [ISSN:2581-5172 (Online)], Eureka Journals.
3. Raj, E., Westerlund, M., and Espinosa-Leal, L. (2021). Reliable Fleet Analytics for Edge IoT Solutions. arXiv.
4. Rai, R.K. (2017). Intricacies of unstructured data. EAI Endorsed Trans. Scalable Inf. Syst.
5. Mohammadi, B., Fathy, M., and Sabokrou, M. (2021). Image/Video Deep Anomaly Detection: A Survey. arXiv.
6. Shrivastava, S., Patel, D., Zhou, N., Iyengar, A., and Bhamidipaty, A. (2020, January 10–13). DQLearn: A Toolkit for Structured Data Quality Learning. Proceedings of the 2020 IEEE International Conference on Big Data (Big Data), Atlanta, GA, USA.10.1109/BigData50022.2020.9378296 ↗
7. Tamburri, D.A. (2020, January 1–4). Sustainable MLOps: Trends and Challenges. Proceedings of the 2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), Timisoara, Romania.10.1109/SYNASC51798.2020.00015 ↗
8. Fursin, G., Guillou, H., and Essayan, N. (2020). CodeReef: An open platform for portable MLOps, reusable automation actions and reproducible benchmarking. arXiv.
9. Granlund, T., Kopponen, A., Stirbu, V., Myllyaho, L., and Mikkonen, T. (2021). MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Cases. arXiv.10.1109/WAIN52551.2021.00019 ↗
10. Zhao, Y. (2021, July 27). Machine Learning in Production: A Literature Review. Available online: https://staff.fnwi.uva.nl/a.s.z.belloum/LiteratureStudies/Reports/2021-LiteratureStudy-report-Yizhen.pdf.
11. Muralidhar, N., Muthiah, S., Butler, P., Jain, M., Yu, Y., Burne, K., Li, W., Jones, D., Arunachalam, P., and McCormick, H.S. (2021). Using AntiPatterns to avoid MLOps Mistakes. arXiv.
12. Silva, L.C., Zagatti, F.R., Sette, B.S., dos Santos Silva, L.N., Lucrédio, D., Silva, D.F., and de Medeiros Caseli, H. (2020, January 14–17). Benchmarking Machine Learning Solutions in Production. Proceedings of the 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), Miami, FL, USA.
13. A Framework for Monitoring Data Warehousing Applications
14. Shivakumar, S.K. (2020). Web Performance Monitoring and Infrastructure Planning. Modern Web Performance Optimization: Methods, Tools, and Patterns to Speed Up Digital Platforms, Apress.10.1007/978-1-4842-6528-4 ↗
15. Sebastian-Coleman, L. (2013). Chapter 4—Data Quality and Measurement. Measuring Data Quality for Ongoing Improvement, Morgan Kaufmann.10.1016/B978-0-12-397033-6.00020-1 ↗
16. Automating large-scale data quality verification10.14778/3229863.3229867 ↗
17. Taleb, I., Serhani, M.A., and Dssouli, R. (2018, January 2–7). Big data quality: A survey. Proceedings of the 2018 IEEE International Congress on Big Data (BigData Congress), San Francisco, CA, USA.10.1109/BigDataCongress.2018.00029 ↗
18. Barrak, A., Eghan, E.E., and Adams, B. (2021, January 9–12). On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects. Proceedings of the 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Honolulu, HI, USA.10.1109/SANER50967.2021.00046 ↗
19. Ramasubramanian, K., and Singh, A. (2017). Machine learning model evaluation. Machine Learning Using R, Apress.10.1007/978-1-4842-2334-5 ↗
20. Concept drift adaptation techniques in distributed environment for real-world data streams10.3390/smartcities4010021 ↗
21. Hutter, F., Kotthoff, L., and Vanschoren, J. (2018). Automated Machine Learning: Methods, Systems, Challenges, Springer. in press. Available online: http://automl.org/book.10.1007/978-3-030-05318-5 ↗
22. Zöller, M.A., and Huber, M.F. (2019). Survey on automated machine learning. arXiv.
23. Peng, G., Lacagnina, C., Downs, R.R., Ramapriyan, H., Ivánová, I., Ganske, A., Jones, D., Bastin, L., Wyborn, L., and Bastrakova, I. (2021, August 27). International Community Guidelines for Sharing and Reusing Quality Information of Individual Earth Science Datasets. OSF Preprints, 16 April 2021. Available online: https://osf.io/xsu4p.10.1002/essoar.10508601.1 ↗
24. Raj, E. (2021). Engineering MLOps: Rapidly Build, Test, and Manage Production-Ready Machine Learning Life Cycles at Scale, Packt Publishing.
25. Wang, D., Liao, Q.V., Zhang, Y., Khurana, U., Samulowitz, H., Park, S., Muller, M., and Amini, L. (2021). How Much Automation Does a Data Scientist Want?. arXiv.
26. (2021, September 14). AWS MLOps Framework. Available online: https://aws.amazon.com/solutions/implementations/aws-mlops-framework/.
27. Sharma, S. (2017). The DevOps Adoption Playbook: A Guide to Adopting DevOps in a Multi-Speed IT Enterprise, John Wiley & Sons.10.1002/9781119310778 ↗
28. Karamitsos, I., Albarhami, S., and Apostolopoulos, C. (2020). Applying DevOps practices of continuous automation for machine learning. Information, 11.10.3390/info11070363 ↗
29. Mäkinen, S., Skogström, H., Laaksonen, E., and Mikkonen, T. (2021). Who Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?. arXiv.10.1109/WAIN52551.2021.00024 ↗
30. Treveil, M., Omont, N., Stenac, C., Lefevre, K., Phan, D., Zentici, J., Lavoillotte, A., Miyazaki, M., and Heidmann, L. (2020). Introducing MLOps, O’Reilly Media.
31. Learning under concept drift: A review
32. Baylor, D., Haas, K., Katsiapis, K., Leong, S., Liu, R., Menwald, C., Miao, H., Polyzotis, N., Trott, M., and Zinkevich, M. (2019, January 20). Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform. Proceedings of the 2019 USENIX Conference on Operational Machine Learning (OpML 19), Santa Clara, CA, USA.
33. Google (2021, May 03). MLOps: Continuous Delivery and Automation Pipelines in Machine Learning. Available online: https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning/.
34. Maydanchik, A. (2007). Data Quality Assessment, Technics Publications.
35. Verheul, I., Imming, M., Ringerma, J., Mordant, A., Ploeg, J.L.V.D., and Pronk, M. (2021, August 27). Data Stewardship on the Map: A study of Tasks and Roles in Dutch Research Institutes. Available online: https://zenodo.org/record/2669150#.YUw2BH0RVPY.
36. Wende, K. (2007, January 5–7). A model for data governance–Organising accountabilities for data quality management. Proceedings of the Data Stewardship on the Map: A Study of Tasks and Roles in Dutch Research Institutes, Toowoomba, Australia.
37. “Data Stewardship Wizard”: A tool bringing together researchers, data stewards, and data experts around data management planning10.5334/dsj-2019-059 ↗
38. Peng, G., Ritchey, N.A., Casey, K.S., Kearns, E.J., Privette, J.A., Saunders, D., Jones, P., Maycock, T., and Ansari, S. (2021, August 27). Scientific Stewardship in the Open Data and Big Data Era-Roles and Responsibilities of Stewards and Other Major Product Stakeholders. Available online: https://www.dlib.org/dlib/may16/peng/05peng.html.
39. Mons, B. (2018). Data Stewardship for Open Science: Implementing FAIR Principles, CRC Press.10.1201/9781315380711 ↗
40. Cloudy, increasingly FAIR; revisiting the FAIR Data guiding principles for the European Open Science Cloud
41. Zubair, N., Hebbar, K., and Simmhan, Y. (2019). Characterizing IoT data and its quality for use. arXiv.
42. Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations
43. Dong, X.L., and Rekatsinas, T. (2018, January 10–15). Data integration and machine learning: A natural synergy. Proceedings of the 2018 International Conference on Management of Data, Houston, TX, USA.10.1145/3183713.3197387 ↗
44. Streamlining the development of data-driven industrial applications by automated machine learning10.1016/j.procir.2020.04.009 ↗
45. From the Edge to the Cloud: Model Serving in ML. NET
46. Image quality assessment: From error visibility to structural similarity10.1109/TIP.2003.819861 ↗
47. Content-partitioned structural similarity index for image quality assessment10.1016/j.image.2010.03.004 ↗
48. Information content weighting for perceptual image quality assessment10.1109/TIP.2010.2092435 ↗
49. Li, C., and Bovik, A.C. (2009, January 19–21). Three-component weighted structural similarity index. Proceedings of the Image Quality and System Performance VI. International Society for Optics and Photonics, San Jose, CA, USA.10.1117/12.811821 ↗
50. de Freitas Zampolo, R., and Seara, R. (2005, January 11–14). A comparison of image quality metric performances under practical conditions. Proceedings of the IEEE International Conference on Image Processing 2005, Genoa, Italy.10.1109/ICIP.2005.1530611 ↗
51. Image information and visual quality10.1109/TIP.2005.859378 ↗
52. FSIM: A feature similarity index for image quality assessment10.1109/TIP.2011.2109730 ↗
53. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index10.1109/TIP.2013.2293423 ↗
54. Egiazarian, K., Astola, J., Ponomarenko, N., Lukin, V., Battisti, F., and Carli, M. (2006, January 22–24). New full-reference quality metrics based on HVS. Proceedings of the Second International Workshop on Video Processing and Quality Metrics, Scottsdale, AZ, USA.
55. Most apparent distortion: Full-reference image quality assessment and the role of strategy10.1117/1.3267105 ↗
56. Towards a full-reference quality assessment for color images using directional statistics10.1109/TIP.2015.2456419 ↗
57. VSI: A visual saliency-induced index for perceptual image quality assessment10.1109/TIP.2014.2346028 ↗
58. Mattson, P., Cheng, C., Coleman, C., Diamos, G., Micikevicius, P., Patterson, D., Tang, H., Wei, G.Y., Bailis, P., and Bittorf, V. (2019). Mlperf training benchmark. arXiv.
59. (2021, July 27). MLflow. Available online: https://mlflow.org/.
60. (2021, July 27). Polyaxon—Machine Learning at Scale. Available online: https://polyaxon.com/.
61. (2021, July 27). Kedro: A Python Framework for Creating Reproducible, Maintainable and Modular Data Science Code. Available online: https://github.com/quantumblacklabs/kedro.
62. Baylor, D., Breck, E., Cheng, H.T., Fiedel, N., Foo, C.Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., and Koc, L. (2017, January 13–17). Tfx: A tensorflow-based production-scale machine learning platform. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada.10.1145/3097983.3098021 ↗
63. (2021, July 27). ZenML. Available online: https://zenml.io/.
64. (2021, September 02). H2O: Tfully Open Source, Distributed in-Memory Machine Learning Platform. Available online: https://www.h2o.ai/products/h2o/.
65. (2021, July 27). Kubeflow: The Machine Learning Toolkit for Kubernetes. Available online: https://www.kubeflow.org/.
66. (2021, July 27). Flyte: The Workflow Automation Platform for Complex, Mission-Critical Data and ML Processes at Scale. Available online: https://flyte.org/.
67. (2021, July 27). Apache Airflow, a Platform Created by the Community to Programmatically Author, Schedule and Monitor Workflows. Available online: https://airflow.apache.org/.
68. (2021, July 27). DVC: Open-Source Version Control System for Machine Learning Projects. Available online: https://dvc.org/.
69. (2021, July 27). The Data Foundation for Machine Learning. Available online: https://www.pachyderm.com/.
70. (2021, July 27). Quilt. Available online: https://quiltdata.com/.
71. (2021, July 27). Great Expectations. Available online: https://greatexpectations.io/.
72. (2021, July 27). Git Large File Storage (LFS). Available online: https://git-lfs.github.com/.
73. (2021, July 27). Continuous Machine Learning (CML). Available online: https://cml.dev/.
74. (2021, July 27). GitHub Actions. Available online: https://github.com/features/actions.
75. (2021, July 27). circleci. Available online: https://circleci.com/.
76. (2021, July 27). gocd. Available online: https://www.gocd.org/.
77. (2021, July 27). Cortex. Available online: https://www.cortex.dev/.
78. (2021, July 27). Seldon Core. Available online: https://github.com/SeldonIO/seldon-core.
79. (2021, July 27). BentoML. Available online: https://github.com/bentoml/BentoML.
80. (2021, July 27). Prometheus—Monitoring System and Time Series Database. Available online: https://prometheus.io/.
81. (2021, July 27). Kibana. Available online: https://www.elastic.co/kibana/.
82. (2021, July 27). Grafana: The Open Observability Platform. Available online: https://grafana.com.
83. (2021, July 27). Lable Studio. Available online: https://labelstud.io/.
84. (2021, July 27). Make Sense. Available online: https://www.makesense.ai/.
85. Tao, X., Zhang, D., Ma, W., Liu, X., and Xu, D. (2018). Automatic metallic surface defect detection and recognition with convolutional neural networks. Appl. Sci., 8.10.3390/app8091575 ↗
86. Ruff, L., Kauffmann, J.R., Vandermeulen, R.A., Montavon, G., Samek, W., Kloft, M., Dietterich, T.G., and Müller, K.R. (2021, January 28–30). A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, Xiamen, China.10.1109/JPROC.2021.3052449 ↗
87. Ultralytics (2021, July 22). YOLOv5. Available online: https://github.com/ultralytics/yolov5.
88. Bergstra, J., Yamins, D., and Cox, D. (2013, January 16–21). Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. Proceedings of the International Conference on Machine Learning, Atlanta, GA, USA.
89. Ahmed, M., Hashmi, K.A., Pagani, A., Liwicki, M., Stricker, D., and Afzal, M.Z. (2021). Survey and Performance Analysis of Deep Learning Based Object Detection in Challenging Environments. Sensors, 21.10.20944/preprints202106.0590.v1 ↗