An Experimental Analysis of Drift Detection Methods on Multi-Class Imbalanced Data Streams
Source abstract
Abstract
The performance of machine learning models diminishes while predicting the Remaining Useful Life (RUL) of the equipment or fault prediction due to the issue of concept drift. This issue is aggravated when the problem setting comprises multi-class imbalanced data. The existing drift detection methods are designed to detect certain drifts in specific scenarios. For example, the drift detector designed for binary class data may not produce satisfactory results for applications that generate multi-class data. Similarly, the drift detection method designed for the detection of sudden drift may struggle with detecting incremental drift. Therefore, in this experimental investigation, we seek to investigate the performance of the existing drift detection methods on multi-class imbalanced data streams with different drift types. For this reason, this study simulated the streams with various forms of concept drift and the multi-class imbalance problem to test the existing drift detection methods. The findings of current study will aid in the selection of drift detection methods for use in developing solutions for real-time industrial applications that encounter similar issues. The results revealed that among the compared methods, DDM produced the best average F1 score. The results also indicate that the multi-class imbalance causes the false alarm rate to increase for most of the drift detection methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Data Stream Mining Techniques · Anomaly Detection Techniques and Applications · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer and Information Sciences, Universiti Teknologi PETRONAS (UTP), Tronoh 32610, Perak, Malaysia
Anti Narcotics Force, Ministry of Narcotics Control, Rawalpindi 46000, Pakistan
Centre for Research in Data Science, Universiti Teknologi PETRONAS (UTP), Tronoh 32610, Perak, Malaysia
School of Engineering and Computer Science, Victoria University of Wellington, Wellington 6012, New Zealand
AI Institute, University of Waikato, Wellington 3240, New Zealand
High-Performance Cloud Computing Centre (HPC3), Universiti Teknologi PETRONAS (UTP), Tronoh 32610, Perak, Malaysia
Funding
UTP URIF
References (53)
Show source-provided references
1. An improved Wiener process model with adaptive drift and diffusion for online remaining useful life prediction10.1016/j.ymssp.2019.03.019 ↗
2. Brownian motion with adaptive drift for remaining useful life prediction: Revisited10.1016/j.ymssp.2017.07.015 ↗
3. A nonlinear-drift-driven Wiener process model for remaining useful life estimation considering three sources of variability10.1016/j.ress.2021.107631 ↗
4. Sensor failure detection and faulty data accommodation approach for instrumented wastewater infrastructures10.1109/ACCESS.2018.2872506 ↗
5. An integrated ensemble learning model for imbalanced fault diagnostics and prognostics10.1109/ACCESS.2018.2807121 ↗
6. Data-Driven Based Fault Prognosis for Industrial Systems A Concise Overview10.1109/JAS.2019.1911804 ↗
7. Deep learning fault diagnosis method based on global optimization GAN for unbalanced data10.1016/j.knosys.2019.07.008 ↗
8. A novel class imbalance-robust network for bearing fault diagnosis utilizing raw vibration signals10.1016/j.measurement.2020.107567 ↗
9. New imbalanced fault diagnosis framework based on Cluster-MWMOTE and MFO-optimized LS-SVM using limited and complex bearing data10.1016/j.engappai.2020.103966 ↗
10. A new dynamic modeling framework for credit risk assessment10.1016/j.eswa.2015.09.055 ↗
11. Applications of machine learning methods for engineering risk assessment—A review10.1016/j.ssci.2019.09.015 ↗
12. Driving Safety Risk Prediction Using Cost-Sensitive with Nonnegativity-Constrained Autoencoders Based on Imbalanced Naturalistic Driving Data10.1109/TITS.2018.2886280 ↗
13. A predictive analytics framework for identifying patients at risk of developing multiple medical complications caused by chronic diseases10.1016/j.artmed.2019.101750 ↗
14. Machine Learning Readmission Risk Modeling: A Pediatric Case Study10.1155/2019/8532892 ↗
15. Khandekar, V.S., and Srinath, P. (2020, January 16–18). Non-stationary Data Stream Analysis: State-of-the-Art Challenges and Solutions. Proceedings of the International Conference on Computational Science and Applications, Las Vegas, NV, USA.10.1007/978-981-15-0790-8_8 ↗
16. Ensemble of online neural networks for non-stationary and imbalanced data streams10.1016/j.neucom.2013.05.003 ↗
17. Agrahari, S., and Singh, A.K. (2021). Concept Drift Detection in Data Stream Mining: A literature review. J. King Saud Univ.—Comput. Inf. Sci., in press.10.1016/j.jksuci.2021.11.006 ↗
18. Learning under Concept Drift: A Review
19. Online neural network model for non-stationary and imbalanced data stream classification10.1007/s13042-013-0180-6 ↗
20. Population structure-learned classifier for high-dimension low-sample-size class-imbalanced problem10.1016/j.engappai.2022.104828 ↗
21. UnderBagging based reduced Kernelized weighted extreme learning machine for class imbalance learning10.1016/j.engappai.2018.07.002 ↗
22. Korycki, Ł., and Krawczyk, B. (2021). Concept Drift Detection from Multi-Class Imbalanced Data Streams. arXiv.10.1109/ICDE51399.2021.00097 ↗
23. A systematic study of online class imbalance learning with concept drift10.1109/TNNLS.2017.2771290 ↗
24. A systematic review on imbalanced data challenges in machine learning: Applications and solutions
25. BPSO-Adaboost-KNN ensemble learning algorithm for multi-class imbalanced data classification10.1016/j.engappai.2015.09.011 ↗
26. Data imbalance in classification: Experimental evaluation10.1016/j.ins.2019.11.004 ↗
27. Faria, E.R., Gama, J., and Carvalho, A.C. (2013, January 18–22). Novelty detection algorithm for data streams multi-class problems. Proceedings of the 28th Annual ACM Symposium on Applied Computing, Coimbra, Portugal.10.1145/2480362.2480515 ↗
28. Gama, J., Medas, P., Castillo, G., and Rodrigues, P. (2004, January 23–28). Learning with drift detection. Proceedings of the Brazilian Symposium on Artificial Intelligence, São Bernardo do Campo, Brazil.10.1007/978-3-540-28645-5_29 ↗
29. Baena-Garcıa, M., del Campo-Ávila, J., Fidalgo, R., Bifet, A., Gavalda, R., and Morales-Bueno, R. (2006, January 6). Early drift detection method. Proceedings of the Fourth International Workshop on Knowledge Discovery from Data Streams, Pennsylvania, PA, USA.
30. RDDM: Reactive drift detection method10.1016/j.eswa.2017.08.023 ↗
31. Bifet, A., and Gavalda, R. (2007, January 26–28). Learning from time-changing data with adaptive windowing. Proceedings of the 2007 SIAM International Conference on Data Mining, Minneapolis, MN, USA.10.1137/1.9781611972771.42 ↗
32. Information resources estimation for accurate distribution-based concept drift detection10.1016/j.ipm.2022.102911 ↗
33. Dasu, T., Krishnan, S., Venkatasubramanian, S., and Yi, K. (2006, January 24–27). An information-theoretic approach to detecting changes in multi-dimensional data streams. Proceedings of the Interface of Statistics, Computing Science, and Applications, Pasadena, CA, USA.
34. An efficient concept drift detection method for streaming data under limited labeling10.1587/transinf.2017EDP7091 ↗
35. Control chart tests based on geometric moving averages10.1080/00401706.2000.10485986 ↗
36. Continuous inspection schemes10.1093/biomet/41.1-2.100 ↗
37. Mouss, H., Mouss, D., Mouss, N., and Sefouhi, L. (2004, January 20–23). Test of page-hinckley, an approach for fault detection in an agro-alimentary production system. Proceedings of the 2004 5th Asian Control Conference (IEEE Cat. No. 04EX904), Melbourne, Australia.
38. Basseville, M., and Nikiforov, I.V. (1993). Detection of Abrupt Changes: Theory and Application, Prentice Hall Englewood Cliffs.
39. Online and Non-Parametric Drift Detection Methods Based on Hoeffding’s Bounds10.1109/TKDE.2014.2345382 ↗
40. Nishida, K., and Yamauchi, K. (2007, January 1–4). Detecting concept drift using statistical testing. Proceedings of the International Conference on Discovery Science, Sendai, Japan.
41. Reactive soft prototype computing for concept drift streams10.1016/j.neucom.2019.11.111 ↗
42. Berger, V.W., and Zhou, Y. (2022, July 05). Kolmogorov—Smirnov Test: Overview. Available online: https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118445112.stat06558.
43. Concept learning using one-class classifiers for implicit drift detection in evolving data streams
44. A comparative study on concept drift detectors10.1016/j.eswa.2014.07.019 ↗
45. Performance evaluation of concept drift detection techniques in the presence of noise
46. A large-scale comparison of concept drift detectors10.1016/j.ins.2018.04.014 ↗
47. A comprehensive active learning method for multiclass imbalanced data streams with concept drift10.1016/j.knosys.2021.106778 ↗
48. Preprocessed dynamic classifier ensemble selection for highly imbalanced drifted data streams10.1016/j.inffus.2020.09.004 ↗
49. Bifet, A., Holmes, G., Pfahringer, B., Kranen, P., Kremer, H., Jansen, T., and Seidl, T. (2010, January 1–3). Moa: Massive online analysis, a framework for stream classification and clustering. Proceedings of the First Workshop on Applications of Pattern Analysis, Windsor, UK.
50. Hulten, G., Spencer, L., and Domingos, P. (2001, January 26–29). Mining time-changing data streams. Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/502512.502529 ↗
51. Exponentially weighted moving average charts for detecting concept drift10.1016/j.patrec.2011.08.019 ↗
52. A comparison of alternative tests of significance for the problem of m rankings10.1214/aoms/1177731944 ↗
53. Statistical comparisons of classifiers over multiple data sets