Precision, Fitness, Generalization, and Simplicity as Quality Dimensions for Decision Discovery Algorithms
Source abstract
Abstract
Operational decisions significantly influence organizational performance and individual well-being. Decision mining offers a method to discover and analyze decision logic from decision logs, enhancing decision-making processes. However, evaluating the quality of decision discovery algorithms remains a challenge. While precision, fitness, generalization, and simplicity are well-established quality dimensions in process mining, their adaptation to the decision mining domain is underexplored. This study adapts these four dimensions to the necessary characteristics of decision models, providing a framework for evaluating decision discovery algorithms. Using a design science research approach, we develop tailored metrics and functions and demonstrate their application through a practical example of environmental permit management modeled in Decision Model and Notation (DMN). Precision measures how the discovered decision model reproduces the observed fact types and values from the decision log, detecting over-specification in the decision model. Fitness evaluates how completely the decision model covers the behavior in the decision log, identifying missing or under-specified elements in the decision model. Generalization assesses the model’s robustness to unseen decision cases by quantifying how well the discovered decision model performs beyond the training data. Simplicity captures the complexity in the discovered decision model in relation to a human actor-specified threshold. These insights guide decision model improvements, contributing to higher transparency, accountability, and fairness in operational decision-making processes. This research bridges a gap in the body of knowledge by providing a concrete methodology for evaluating decision discovery algorithms. The results support organizations in aligning decision models with regulatory requirements and public values, while also laying a foundation for future research.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Big Data and Business Intelligence
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Research Group Digital Ethics, HU University of Applied Sciences Utrecht, 3584 CH Utrecht, The Netherlands
References (46)
Show source-provided references
1. Mircea, M., Ghilic-Micu, B., and Stoic, M. (2012). An agile architecture framework that leverages the strengths of business intelligence, decision management and service orientation. Business Intelligence, Solution for Business Development.10.5772/35477 ↗
2. Rula, A., Maurino, A., and Batini, C. (2016). Data and Information Quality: Dimensions, Principles and Techniques, Springer. [1st ed.].
3. Sustainable resource allocation for power generation: The role of big data in enabling interindustry architectural innovation10.1016/j.techfore.2018.04.031 ↗
4. (2019). The Object Management Group (OMG) Decision model and notation (DMN). Decis. Model Not. DMN, 1, Available online: http://www.omg.org/spec/DMN/.
5. van der Aalst, W. (1998). Three good reasons for using a petri-net-based workflow management system. Information and Process Integration in Enterprises, Springer.10.1007/978-1-4615-5499-8_10 ↗
6. Boyer, J., and Mili, H. (2011). Agile business rule development. Agile Business Rule Development, Springer.10.1007/978-3-642-19041-4 ↗
7. Smit, K. (2018). Organization and Governance of Business Rules Management Capabilities, Open Universiteit.
8. Batoulis, K., Meyer, A., Bazhenova, E., Decker, G., and Weske, M. (2015, January 8–12). Extracting decision logic from process models. Proceedings of the International Conference on Advanced Information Systems Engineering, Stockholm, Sweden.10.1007/978-3-319-19069-3_22 ↗
9. Deriving decision models from process models by enhanced decision mining
10. Decision mining in a broader context: An overview of the current landscape and future directions10.1007/978-3-319-58457-7_15 ↗
11. Dustdar, S., Fiadeiro, J.L., and Sheth, A.P. (2006). Decision mining in ProM. Business Process Management: 4th International Conference, BPM 2006, Springer.10.1007/11841760 ↗
12. Leewis, S. (2025). Improving Operational Decision-Making Through Decision Mining. [Doctoral Dissertation, Open Universiteit].10.1016/j.infsof.2024.107627 ↗
13. The Object Management Group (2025, October 08). Business Process Model and Notation (BPMN) 2.0. Available online: https://www.omg.org/spec/BPMN.
14. Qiu, H., Zhang, C., Fei, Z., Qiu, M., and Kung, S.Y. (2021). An Overview of Methods for Acquiring and Generating Decision Models. Knowledge Science, Engineering and Management. KSEM 2021, Springer. Lecture Notes in Computer Science.10.1007/978-3-030-82136-4 ↗
15. Dasseville, I., Janssens, L., Janssens, G., Vanthienen, J., and Denecker, M. (2016, January 1). Combining DMN and the knowledge base paradigm for flexible decision enactment. Proceedings of the RuleML 2016 Challenge, Doctoral Consortium and Industry Track Hosted by the 10th International Web Rule Symposium (RuleML 2016), Stony Brook, NY, USA. Available online: https://ceur-ws.org/Vol-1620/paper3.pdf.
16. Devine, P.W., Srinivasan, C.A., and Zaman, M.S. (2004). Importance of data in decision-making. Business Intelligence Techniques, Springer.10.1007/978-3-540-24700-5_2 ↗
17. Halle, B., and Goldberg, L. (2009). The Decision Model: A Business Logic Framework Linking Business and Technology, Taylor and Francis Group, LLC.
18. Just accountability structures—A way to promote the safe use of automated decision-making in the public sector10.1007/s00146-023-01731-z ↗
19. Koene, A., Clifton, C., Hatada, Y., Webb, H., and Richardson, R. (2019). A governance framework for algorithmic accountability and transparency. EPRS Eur. Parliam. Res. Serv., 1–124.
20. Fair, transparent, and accountable algorithmic decision-making processes10.1007/s13347-017-0279-x ↗
21. Algorithms and decision-making in the public sector10.1146/annurev-lawsocsci-041221-023808 ↗
22. Veale, M., Kleek, M., and Binns, R. (2018, January 21–26). Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.10.1145/3173574.3174014 ↗
23. Applying the FAIR Principles to computational workflows10.1038/s41597-025-04451-9 ↗
24. van der Aalst, W.M.P. (2011). Process Mining, Springer.10.1007/978-3-642-19345-3 ↗
25. On the role of fitness, precision, generalization and simplicity in process discovery
26. Quality dimensions in process discovery: The importance of fitness, precision, generalization and simplicity10.1142/S0218843014400012 ↗
27. Replaying history on process models for conformance checking and performance analysis10.1002/widm.1045 ↗
28. Optimal scale combination selection based on genetic algorithm in generalized multi-scale decision systems for classification10.1016/j.ins.2024.121685 ↗
29. Optimal scale selection and knowledge discovery in generalized multi-scale decision tables10.1016/j.ijar.2023.108983 ↗
30. Positioning and presenting design science research for maximum impact10.25300/MISQ/2013/37.2.01 ↗
31. Comparing conformance checking for decision mining: An axiomatic approach10.1109/ACCESS.2024.3391234 ↗
32. Wais, B., and Rinderle-Ma, S. (2024, January 3–7). Towards a comprehensive evaluation of decision rules and decision mining algorithms beyond accuracy. Proceedings of the International Conference on Advanced Information Systems Engineering, Limassol, Cyprus.10.1007/978-3-031-61057-8_24 ↗
33. Discovering injury severity risk factors in automobile crashes: A hybrid explainable AI framework for decision support10.1016/j.ress.2022.108720 ↗
34. Han, J., Kamber, M., and Pei, J. (2012). Data Mining: Concepts and Techniques: Concepts and Techniques, Morgan Kaufmann. [3rd ed.].
35. Smedt, J., Hasić, F., Broucke, S.K.L.M., and Vanthienen, J. (2017, January 10–15). Towards a holistic discovery of decisions in process-aware information systems. Proceedings of the International Conference on Business Process Management, Barcelona, Spain.
36. Mannhardt, F., Leoni, M., Reijers, H.A., and van der Aalst, W.M.P. (2016, January 13–17). Decision mining revisited-discovering overlapping rules. Proceedings of the International Conference on Advanced Information Systems Engineering, Ljubljana, Slovenia.10.1007/978-3-319-39696-5_23 ↗
37. Scheibel, B., and Rinderle-Ma, S. (2022). Decision Mining with Time Series Data Based on Automatic Feature Generation, Springer.10.1007/978-3-031-07472-1_1 ↗
38. van der Aalst, W.M.P. (2013, January 29–31). Mediating between modeled and observed behavior: The quest for the “right” process: Keynote. Proceedings of the IEEE 7th International Conference on Research Challenges in Information Science (RCIS), Paris, France.10.1109/RCIS.2013.6577675 ↗
39. Petri, C.A. (1962). Kommunikation mit Automaten (Communication with Automata). [Ph.D. Thesis, University of Bonn].
40. Morgan, T. (2002). Business Rules and Information Systems: Aligning IT with Business Goals, Addison-Wesley.
41. Putting decision mining into context: A literature study10.1007/978-3-030-47355-6_3 ↗
42. The Object Management Group (OMG) (2025, October 08). Semantics of Business Vocabulary and Business Rules (SBVR) 2013, 1 4. Available online: https://www.omg.org/spec/SBVR/1.4.
43. Design science in information systems research10.2307/25148625 ↗
44. Complexity metrics for DMN decision models10.1016/j.csi.2019.01.006 ↗
45. De Leoni, M., and van der Aalst, W.M.P. (2013, January 18–22). Data-aware process mining: Discovering decisions in processes using alignments. Proceedings of the 28th Annual ACM Symposium on Applied Computing, Coimbra, Portugal.10.1145/2480362.2480633 ↗
46. Management control system for business rules management