journal article·2020-04-25Open access

Self-Service Data Science in Healthcare with Automated Machine Learning

Richard Ooms, Marco Spruit

Applied Sciences· Volume 10· Issue 9

Abstract

(1) Background: This work investigates whether and how researcher-physicians can be supported in their knowledge discovery process by employing Automated Machine Learning (AutoML). (2) Methods: We take a design science research approach and select the Tree-based Pipeline Optimization Tool (TPOT) as the AutoML method based on a benchmark test and requirements from researcher-physicians. We then integrate TPOT into two artefacts: a web application and a notebook. We evaluate these artefacts with researcher-physicians to examine which approach suits researcher-physicians best. Both artefacts have a similar workflow, but different user interfaces because of a conflict in requirements. (3) Results: Artefact A, a web application, was perceived as better for uploading a dataset and comparing results. Artefact B, a Jupyter notebook, was perceived as better regarding the workflow and being in control of model construction. (4) Conclusions: Thus, a hybrid artefact would be best for researcher-physicians. However, both artefacts missed model explainability and an explanation of variable importance for their created models. Hence, deployment of AutoML technologies in healthcare remains currently limited to the exploratory data analysis phase.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Scientific Computing and Data Management · Machine Learning and Data Classification · Artificial Intelligence in Healthcare

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Information and Computing Sciences, Utrecht University, 3512 JE Utrecht, The Netherlands

References (58)

Show source-provided references
1. A Systematic Review on Healthcare Analytics: Application and Theoretical Perspective of Data Mining10.3390/healthcare6020054 ↗
2. Bringing big data to personalized healthcare: A patient-centered framework10.1007/s11606-013-2455-8 ↗
3. Big data analytics in healthcare: Promise and potential
4. Applied data science in patient-centric healthcare: Adaptive analytic systems for empowering physicians and patients10.1016/j.tele.2018.04.002 ↗
5. Spruit, M., and Vries de, N. (2020, January 15–17). Self-Service Data Science for Adverse Event Prediction with AutoCrisp. Proceedings of the Springer Proceedings in Complexity, Research & Innovation Forum, Athens, Greece.
6. Spruit, M., and Jagesar, R. (2016, January 9–11). Power to the People!—Meta-Algorithmic Modelling in Applied Data Science. Proceedings of the 8th International Conference on Knowledge Discovery and Information Retrieval, Porto, Portugal.10.5220/0006081604000406 ↗
7. Spruit, M.R., Dedding, T., and Vijlbrief, D. (2020, January 24–26). Self-Service Data Science for Healthcare Professionals: A Data Preparation Approach. Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020)—Volume 5: HEALTHINF, Valletta, Malta.10.5220/0009169500002513 ↗
8. Thornton, C., Hutter, F., Hoos, H.H., and Leyton-Brown, K. (2013, January 22–27). Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA.10.1145/2487575.2487629 ↗
9. Hutter, F., Kotthoff, L., and Vanschoren, J. (2019). Automatic Machine Learning: Methods, Systems, Challenges, Springer.10.1007/978-3-030-05318-5 ↗
10. Zhang, Y., Bahadori, M.T., Su, H., and Sun, J. (2016, January 13–17). FLASH: Fast Bayesian Optimization for Data Analytic Pipelines. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939829 ↗
11. Alaa, A.M., and van der Schaar, M. (2018, January 10–15). AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden.
12. Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA
13. Komer, B., Bergstra, J., and Eliasmith, C. (2014). Hyperopt-Sklearn: Automatic HyperparameterConfiguration for Scikit-Learn. ICML Work. AutoML, 2825–2830.10.25080/Majora-14bd3278-006 ↗
14. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J.T., Blum, M., and Hutter, F. (2015, January 7–12). Efficient and Robust Automated Machine Learning. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
15. Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., and Hutter, F. (2018, January 10–15). Practical Automated Machine Learning for the AutoML Challenge 2018. Proceedings of the ICML 2018 AutoML Workshop, Stockholm, Sweden.
16. ML-Plan: Automated machine learning via hierarchical planning10.1007/s10994-018-5735-z ↗
17. Bergstra, J.S., Bardenet, R., Bengio, Y., and Kégl, B. (2011, January 12–14). Algorithms for hyper-parameter optimization. Proceedings of the Advances in Neural Information Processing Systems, Granada, Spain.
18. Bergstra, J., Yamins, D., and Cox, D.D. (2013). Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms. Comput. Sci. Discov., 1–8.10.25080/Majora-8b375195-003 ↗
19. Vanschoren, J., van Rijn, J.N., Bischl, B., and Torgo, L. (2014). OpenML: Networked science in machine learning. ACM SIGKDD Explor. Newsl.10.1145/2641190.2641198 ↗
20. SHOP2: An HTN planning system10.1613/jair.1141 ↗
21. Towards Automatically-Tuned Neural Networks
22. Drori Krishnamurthy, Y., Rampin, R., Lourenço, R., Ono, J.P., Cho, K., Silva, C.T., and Freire, J. (2018, January 10–15). AlphaD3M: Machine Learning Pipeline Synthesis. Proceedings of the ICML 2018 AutoML Workshop, Stockholm, Sweden.
23. Jin, H., Song, Q., and Hu, X. (2018). Auto-Keras: Efficient Neural Architecture Search with Network Morphism. arXiv.10.1145/3292500.3330648 ↗
24. Olson, R.S., and Moore, J.H. (2006). TPOT: A tree-based pipeline optimization tool for automating machine learning. Work. Automated Machine Learning, Springer.
25. Gijsbers, P., Vanschoren, J., and Olson, R.S. (2017, January 18–22). Layered TPOT: Speeding up tree-based pipeline optimization. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Skopje, Macedonia.
26. RECIPE: A grammar-based framework for automatically evolving classification pipelines
27. Chen, B., Wu, H., Mo, W., Chattopadhyay, I., and Lipson, H. (2018, January 15–19). Autostacker: A Compositional Evolutionary Learning System. Proceedings of the Genetic and Evolutionary Computation Conference, Kyoto, Japan.10.1145/3205455.3205586 ↗
28. Hornby, G.S. (2006, January 8–12). ALPS: The Age-Layered Population Structure for Reducing the Problem of Premature Convergence. Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, Seattle, DC, USA.10.1145/1143997.1144142 ↗
29. Swearingen, T., Drevo, W., Cyphers, B., Cuesta-Infante, A., Ross, A., and Veeramachaneni, K. (2017, January 11–14). ATM: A distributed, collaborative, scalable system for automated machine learning. Proceedings of the 2017 IEEE International Conference on Big Data, Big Data 2017, Boston, MA, USA.10.1109/BigData.2017.8257923 ↗
30. Design Science in Information Systems10.2307/25148625 ↗
31. Gijsbers, P., Ledell, E., Thomas, J., Poirier, S., Bischl, B., and Vanschoren, J. (2019, January 14). An Open Source AutoML Benchmark. Proceedings of the ICML Workshop on AutoML, Long Beach, CA, USA.
32. Bischl, B., Casalicchio, G., Feurer, M., Hutter, F., Lang, M., Mantovani, R.G., van Rijn, J.N., and Vanschoren, J. (2017). OpenML Benchmarking Suites and the OpenML100. arxiv.
33. Cancer Diagnosis via Linear Programming
34. Dua, D., and Graff, C. (2019, April 28). UCI Machine Learning Repository. Available online: https://archive.ics.uci.edu/ml/index.php.
35. Simplifying decision trees. International journal of man-machine studies10.1016/S0020-7373(87)80053-6 ↗
36. Davis, A., Dieste, O., Hickey, A., Juristo, N., and Moreno, A.M. (2006, January 11–15). Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review. Proceedings of the 14th IEEE International Requirements Engineering Conference, Minneapolis/St. Paul, MN, USA.10.1109/RE.2006.17 ↗
37. Qualitative research and its uses in health care
38. Qualitative methods in research on healthcare quality10.1136/qhc.11.2.148 ↗
39. Preparing for interview research: The interview protocol refinement framework
40. FEDS: A Framework for Evaluation in Design Science Research10.1057/ejis.2014.36 ↗
41. Offermann, P., Levina, O., Schönherr, M., and Bub, U. (2009, January 6–8). Outline of a Design Science Research Project. Proceedings of the 4th International Conference Design Science Research in Information Systems and Technologies, Philadelphia, PA, USA.10.1145/1555619.1555629 ↗
42. Cohen, M. (2004). User Stories Applied: For Agile Software Development, Addison-Wesley Professional.
43. Janssen, J.H.N. (2019). The Right to Explanation: Means for ‘White-Boxing’ the Black-Box?. [Master’s Thesis, Tilburg University].
44. Chapman, P., Clinton, J., Kerber, R., Khabaza, T., Reinartz, T., Shearer, C., and Wirth, R. (1999, January 18). The CRISP-DM user guide. Proceedings of the 4th CRISP-DM SIG Workshop, Brussels, Belgium.
45. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models10.1016/j.jclinepi.2019.02.004 ↗
46. Gareth, J., Witten, D., Hastie, T., and Tibshirani, R. (2013). An Introduction to Statistical Learning with Applications in R, Springer.
47. Supervised Machine Learning: A Review of Classification Techniques
48. Vollmer, S., Mateen, B., Bohner, G., Király, F., Ghani, R., Jonsson, P., Cumbers, S., Jonas, A., McAllister, K., and Myles, P. (2018). ine learning and AI research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics and Effectiveness. arXiv.
49. Kaplan, B., and Maxwell, J. (2005). Qualitative Research Methods for Evaluating Computer Information Systems. Evaluating the Organizational Impact of Healthcare Information Systems, Springer.10.1007/0-387-30329-4_2 ↗
50. Examining the Validity Structure of Qualitative Research
51. Kiegelmann, M. (2002). The Role of the Researcher in Qualitative Psychology, Ingeborg Huber Verlag.
52. Generalization in quantitative and qualitative research: Myths and strategies10.1016/j.ijnurstu.2010.06.004 ↗
53. Michael, W.D., Dustin, T., Edward, C., Jonas, K., Jeremy, N., Ghassen, J., Katherine, H., and Andrew, M.D. (2019). Analyzing the Role of Model Uncertainty for Electronic Health Records. arXiv.
54. Central Challenges Facing the National Clinical Research Enterprise10.1001/jama.289.10.1278 ↗
55. Molnar, C. (2019). Interpretable Machine Learning, Leanpub.10.21105/joss.00786 ↗
56. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 23). Model-Agnostic Interpretability of Machine Learning. Proceedings of the ICML Workshop on Human Interpretability in Machine Learning, New York, NY, USA.
57. Shapley, L.S. (1953). A value for n-person games. Contributions to the Theory of Games, Princeton University Press.10.1515/9781400881970-018 ↗
58. Bartoletti, I. (2019). AI in Healthcare: Ethical and Privacy Challenges. Artificial Intelligence in Medicine, Elsevier.10.1007/978-3-030-21642-9_2 ↗