Metrics Related to Confusion Matrix as Tools for Conformity Assessment Decisions
Source abstract
Abstract
Conformity assessment refers to activities undertaken to check whether some product, service or process meets certain criteria and specifications given by internationally accepted standards. The decision on whether a property of interest is aligned with the set standards is made based on measurement. However, uncertainty associated with the measurement results may lead to incorrect decisions. Measurement results may be falsely rejected as non-conforming, although they meet specifications. This is referred to as the producer’s risk. If the measurement result that does not meet the required specifications is accepted as conforming, this is referred to as the consumer’s risk. This paper covers calculations of global consumer's and producer's risk using the Bayesian approach and deals with the application of metrics related to confusion matrices in conformity assessments. These techniques have been used to assess the conformity of the bearing ring diameter with the given specifications. Based on the behavior of these metrics, the optimal length of the guard band was determined with the aim of minimizing the global consumer’s and producer’s risk.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Statistical Process Monitoring · Advanced Statistical Methods and Models · Pesticide Residue Analysis and Safety
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Quality, Faculty of Mechanical Engineering and Naval Architecture, University of Zagreb, 10000 Zagreb, Croatia
Department of Industrial Engineering and Management, Faculty of Mechanical Engineering and Naval Architecture, University of Zagreb, 10000 Zagreb, Croatia
Funding
European Regional Development Fund
References (50)
Show source-provided references
1. BIPM, IEC, IFCC, ILAC, ISO, IUPAC, IUPAP, and OIML (2023, March 26). Evaluation of Measurement Data—The Role of Measurement Uncertainty in Conformity Assessment, JCGM 106:2012. BIPM. Available online: https://www.bipm.org/documents/20126/2071204/JCGM_106_2012_E.pdf/fe9537d2-e7d7-e146-5abb-2649c3450b25?version=1.7&t=1659083025736&download=true.
2. Allard, A., Fischer, N., Smith, I., Harris, P., and Pendrill, L. (2019, January 24–26). Risk calculations for conformity assessment in practice. Proceedings of the 19th International Congress of Metrology, Paris, France.10.1051/metrology/201916001 ↗
3. Runje, B., Horvatić Novak, A., Razumić, A., Piljek, P., Štrbac, B., and Orošnjak, M. (2019, January 23–26). Evaluation of Consumer and Producer Risk in Conformity Assessment Decision. Proceedings of the 30th DAAAM International Symposium “Intelligent Manufacturing & Automation”, Zadar, Croatia.10.2507/30th.daaam.proceedings.007 ↗
4. (2023, May 18). ILAC-G8:09/2019. Guidelines on Decision Rules and Statements of Conformity. Available online: https://ilac.org/publications-and-resources/ilac-guidance-series/.
5. Puydarrieux, S., Pou, J.M., Leblond, L., Fischer, N., Allard, A., Feinberg, M., and El Guennouni, D. (2019, January 24–26). Role of measurement uncertainty in conformity assessment. Proceedings of the 19th International Congress of Metrology, Paris, France.10.1051/metrology/201916003 ↗
6. Measurement uncertainty evaluation and risk of false conformity assessment for microbial enumeration tests10.1016/j.mimet.2021.106312 ↗
7. Bettencourt da Silva, R. (2023, May 18). Eurachem/CITAC Guide: Setting and Using Target Uncertainty in Chemical Measurement. Available online: https://www.eurachem.org/index.php/publications/guides/gd-stmu.
8. Using measurement uncertainty in decision-making and conformity assessment10.1088/0026-1394/51/4/S206 ↗
9. Williams, A., and Magnusson, B. (2023, May 18). Eurachem/CITAC Guide: Use of Uncertainty Information in Compliance Assessment. Available online: https://www.eurachem.org/index.php/publications/guides/uncertcompliance.
10. (2023, May 18). EUROLAB Technical Report, No. 1/2017-Decision Rules Applied to Conformity Assessment. Available online: https://www.eurolab.org/pubs-techreports.
11. A Bayesian approach to the consumer’s and producer’s risks in measurement10.1088/0026-1394/36/5/1 ↗
12. Toczek, W., and Smulko, J. (2021). Risk Analysis by a Probabilistic Model of the Measurement Process. Sensors, 21.10.3390/s21062053 ↗
13. Conformity Decisions Based on Measurement Uncertainty—A Case Study Applied to Agar Diffusion Microbiological Assay10.1007/s12247-019-09374-8 ↗
14. BIPM, IEC, IFCC, ILAC, ISO, IUPAC, IUPAP, and OIML (2023, March 26). Evaluation of Measurement Data—Supplement 1 to the “Guide to the Expression of Uncertainty in Measurement”—Propagation of Distributions Using a Monte Carlo Method, JCGM 101:2008. BIPM. Available online: https://www.bipm.org/documents/20126/2071204/JCGM_101_2008_E.pdf/325dcaad-c15a-407c-1105-8b7f322d651c?version=1.12&t=1659082897489&download=true.
15. Božić, D., and Runje, B. (2022, January 26–27). Selection of an Appropriate Prior Distribution in Risk Assessment. Proceedings of the 33rd International DAAAM Virtual Symposium “Intelligent Manufacturing & Automation”, Vienna, Austria.10.2507/33rd.daaam.proceedings.066 ↗
16. Specific risks of false decisions in conformity assessment of a substance or material with a mass balance constraint—A case study of potassium iodate10.1016/j.measurement.2020.108662 ↗
17. Determination of intrinsic and metrological components of the correlation of multiparameter products for minimising the risks of false conformity decisions10.1016/j.measurement.2021.109531 ↗
18. Measurement uncertainty and risk of false conformity decision in the performance evaluation of liquid chromatography analytical procedures10.1016/j.jpba.2019.04.005 ↗
19. Conformity assessment of medicines containing antibiotics—A multivariate assessment10.1016/j.yrtph.2022.105279 ↗
20. Conformity assessment of multicomponent materials or objects: Risk of false decisions due to measurement uncertainty—A case study of denatured alcohols10.1016/j.talanta.2016.11.035 ↗
21. Risk of false conformity assessment applied to automotive fuel analysis: A multiparameter approach10.1016/j.chemosphere.2020.128265 ↗
22. Risks in a sausage conformity assessment due to measurement uncertainty, correlation and mass balance constraint10.1016/j.foodcont.2021.107949 ↗
23. Božić, D., Samardžija, M., Kurtela, M., Keran, Z., and Runje, B. (2023). Risk Evaluation for Coating Thickness Conformity Assessment. Materials, 16.10.3390/ma16020758 ↗
24. Brandão, L.P., Silva, V.F., Bassi, M., and de Oliveira, E.C. (2022). Risk Assessment in Monitoring of Water Analysis of a Brazilian River. Molecules, 27.10.3390/molecules27113628 ↗
25. A Bayesian interpretation of the confusion matrix10.1007/s10472-017-9564-8 ↗
26. Grandini, M., Bagli, E., and Visani, G. (2020). Metrics for Multi-Class Classification: An Overview. arXiv.
27. Performance measures in evaluating machine learning based bioinformatics predictors for classifications10.1007/s40484-016-0081-2 ↗
28. The impact of class imbalance in classification performance metrics based on the binary confusion matrix10.1016/j.patcog.2019.02.023 ↗
29. Jeni, L.A., Cohn, J.F., and De La Torre, F. (2013, January 2–5). Facing Imbalanced Data-Recommendations for the Use of Performance Metrics. Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII), Geneva, Switzerland.10.1109/ACII.2013.47 ↗
30. Naive Bayesian Classification of Structured Data10.1023/B:MACH.0000039778.69032.ab ↗
31. Classification assessment methods10.1016/j.aci.2018.08.003 ↗
32. Alvarez, S.A. (2023, May 19). An Exact Analytical Relation among Recall, Precision, and Classification Accuracy in Information Retrieval. Technical Report BC-CS-2002-01. Available online: http://www.cs.bc.edu/~alvarez/APR/aprformula.pdf.
33. Precision-recall-gain curves: PR analysis done right
34. Interrater reliability: The kappa statistic10.11613/BM.2012.031 ↗
35. The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation10.1186/s13040-021-00244-z ↗
36. Chicco, D., and Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genom., 21.10.1186/s12864-019-6413-7 ↗
37. Juba, B., and Le, H.S. (February, January 27). Precision-Recall versus Accuracy and the Role of Large Data Sets. Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.10.1609/aaai.v33i01.33014039 ↗
38. Classification of imbalanced data: A review10.1142/S0218001409007326 ↗
39. Boosting methods for multi-class imbalanced data classification: An experimental review10.1186/s40537-020-00349-y ↗
40. Delgado, R., and Tibau, X.-A. (2019). Why Cohen’s Kappa should be avoided as performance measure in classification. PLoS ONE, 14.10.1371/journal.pone.0222916 ↗
41. Volodarsky, E.T., Kosheva, L.O., and Klevtsova, M.O. (2019, January 6–9). The Role Uncertainty of Measurements in the Formation of Acceptance Criteria. Proceedings of the XXIX International Scientific Symposium “Metrology and Metrology Assurance” (MMA), Sozopol, Bulgaria.10.1109/MMA.2019.8935988 ↗
42. Monitoring fuel quality: A case study for quinizarin marker content of unleaded petrol marketed in Greece10.1007/s00769-016-1199-7 ↗
43. Guard-Band Strategy for Managing False-Accept Risk10.1080/19315775.2008.11721446 ↗
44. Deaver, D. (1994, January 1). Guardbanding with Confidence. Proceedings of the NCSL Workshop and Symposium, Chicago, IL, USA. Available online: https://download.flukecal.co/pub/literature/ddncsl94.pdf.
45. Purata-Sifuentes, O.-J., Hernández-Balandrán, L.-E., and Tornero-Navarro, M.-G. (2022, January 16–20). Role of the measurement uncertainty in cone penetration test results of lubricating grease. Proceedings of the Joint IMEKO TC11 & TC24 hybrid conference, Dubrovnik, Croatia.10.21014/tc11-2022.16 ↗
46. Economic optimization of acceptance interval in conformity assessment: 1. Process with no systematic effect10.1088/1681-7575/ac6fa1 ↗
47. Economic optimization of acceptance interval in conformity assessment: 2. Process with unknown systematic effect10.1088/1681-7575/ac6fa2 ↗
48. Definition of multivariate acceptance limits (guard-bands) applied to pharmaceutical equivalence assessment10.1016/j.jpba.2022.115080 ↗
49. Setting Multivariate and Correlated Acceptance Limits for Assessing the Conformity of Items10.1080/00032719.2022.2042549 ↗
50. The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment10.1109/ACCESS.2021.3084050 ↗