Concept Drift Adaptation with Incremental–Decremental SVM
Source abstract
Abstract
Data classification in streams where the underlying distribution changes over time is known to be difficult. This problem—known as concept drift detection—involves two aspects: (i) detecting the concept drift and (ii) adapting the classifier. Online training only considers the most recent samples; they form the so-called shifting window. Dynamic adaptation to concept drift is performed by varying the width of the window. Defining an online Support Vector Machine (SVM) classifier able to cope with concept drift by dynamically changing the window size and avoiding retraining from scratch is currently an open problem. We introduce the Adaptive Incremental–Decremental SVM (AIDSVM), a model that adjusts the shifting window width using the Hoeffding statistical test. We evaluate AIDSVM performance on both synthetic and real-world drift datasets. Experiments show a significant accuracy improvement when encountering concept drift, compared with similar drift detection models defined in the literature. The AIDSVM is efficient, since it is not retrained from scratch after the shifting window slides.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Data Stream Mining Techniques · Network Security and Intrusion Detection · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Mathematics and Computer Science, Transilvania University of Braşov, 500036 Braşov, Romania
Xperi Corporation, 500152 Braşov, Romania
Computer Science Department, Central Washington University, Ellensburg, WA 98926, USA
Department of Mathematics and Computer Science, Faculty of Mathematics and Computer Science, Transilvania University of Braşov, 500036 Braşov, Romania
References (50)
Show source-provided references
1. New climate models predict a warming surge10.1126/science.364.6437.222 ↗
2. Gama, J. (2010). Knowledge Discovery from Data Streams, Chapman & Hall/CRC. [1st ed.].10.1201/EBK1439826119 ↗
3. Using multiple windows to track concept drift10.3233/IDA-2004-8103 ↗
4. A survey on concept drift adaptation10.1145/2523813 ↗
5. An overview on concept drift learning10.1109/ACCESS.2018.2886026 ↗
6. Learning under concept drift: A review
7. An adaptive ensemble classifier for mining concept drifting data streams10.1016/j.eswa.2013.05.001 ↗
8. Learning in nonstationary environments: A survey10.1109/MCI.2015.2471196 ↗
9. Alippi, C., Qi, W., and Roveri, M. (2017, January 11–15). Learning in nonstationary environments: A hybrid approach. Proceedings of the International Conference on Artificial Intelligence and Soft Computing, Zakopane, Poland.10.1007/978-3-319-59060-8_63 ↗
10. Reactive Soft Prototype Computing for Concept Drift Streams10.1016/j.neucom.2019.11.111 ↗
11. Multi-category news classification using Support Vector Machine based classifiers10.1007/s42452-020-2266-6 ↗
12. Comparative analysis of hybrid models of firefly optimization algorithm with support vector machines and multilayer perceptron for predicting soil temperature at different depths
13. Dabija, A., Kluczek, M., Zagajewski, B., Raczko, E., Kycko, M., Al-Sulttani, A.H., Tardà, A., Pineda, L., and Corbera, J. (2021). Comparison of Support Vector Machines and Random Forests for Corine Land Cover Mapping. Remote Sens., 13.10.3390/rs13040777 ↗
14. Flores, C., Taramasco, C., Lagos, M.E., Rimassa, C., and Figueroa, R.A. (2021). Feature-Based Analysis for Time-Series Classification of COVID-19 Incidence in Chile: A Case Study. Appl. Sci., 11.10.3390/app11157080 ↗
15. Zhu, H., and Woo, J.H. (2021). Hybrid NHPSO-JTVAC-SVM Model to Predict Production Lead Time. Appl. Sci., 11.10.3390/app11146369 ↗
16. Shabani, S., Samadianfard, S., Sattari, M.T., Mosavi, A., Shamshirband, S., Kmet, T., and Várkonyi-Kóczy, A.R. (2020). Modeling Pan Evaporation Using Gaussian Process Regression K-Nearest Neighbors Random Forest and Support Vector Machines. Comparative Analysis. Atmosphere, 11.10.3390/atmos11010066 ↗
17. Klinkenberg, R., and Joachims, T. (July, January 29). Detecting concept drift with support vector machines. Proceedings of the Seventeenth International Conference on Machine Learning (ICML ’00), Stanford, CA, USA.
18. Learning drifting concepts: Example selection vs. example weighting10.3233/IDA-2004-8305 ↗
19. Concept Drift-Oriented Adaptive and Dynamic Support Vector Machine Ensemble With Time Window in Corporate Financial Risk Prediction10.1109/TSMCA.2012.2224338 ↗
20. Bifet, A., and Gavaldà, R. (2007, January 26–28). Learning from time-changing data with adaptive windowing. Proceedings of the Seventh SIAM International Conference on Data Mining (SDM’07), Minneapolis, MN, USA.10.1137/1.9781611972771.42 ↗
21. An overview of unsupervised drift detection methods10.1002/widm.1381 ↗
22. Elwell, R., and Polikar, R. (2009, January 14–19). Incremental learning in nonstationary environments with controlled forgetting. Proceedings of the 2009 International Joint Conference on Neural Networks, Atlanta, GA, USA.10.1109/IJCNN.2009.5178779 ↗
23. Incremental learning of concept drift in nonstationary environments10.1109/TNN.2011.2160459 ↗
24. A Fast Learn++.NSE classification algorithm based on weighted moving average10.2298/FIL1805737S ↗
25. Adaptive random forests for evolving data stream classification10.1007/s10994-017-5642-8 ↗
26. Gama, J., Medas, P., Castillo, G., and Rodrigues, P.P. (October, January 29). Learning with drift detection. Proceedings of the 17th Brazilian Symposium on Artificial Intelligence (SBIA 2004), Sao Luis, Maranhao, Brazil.
27. Early drift detection method
28. Pesaranghader, A., and Viktor, H.L. (2016, January 19–23). Fast Hoeffding drift detection method for evolving data streams. Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Riva del Garda, Italy.10.1007/978-3-319-46227-1_7 ↗
29. Pesaranghader, A., Viktor, H.L., and Paquet, E. (2016). A framework for classification in data streams using multi-strategy learning. International Conference on Discovery Science, Springer.10.1007/978-3-319-46307-0_22 ↗
30. Reservoir of diverse adaptive learners and stacking fast Hoeffding drift detection methods for evolving data streams10.1007/s10994-018-5719-z ↗
31. Pesaranghader, A. (2018). A Reservoir of Adaptive Algorithms for Online Learning from Evolving Data Streams. [Ph.D. Dissertation, University of Ottawa].
32. Accurate detecting concept drift in evolving data streams10.1016/j.icte.2020.05.011 ↗
33. ZareMoodi, P., Siahroudi, S.K., and Beigy, H. (2016, January 4–8). A support vector based approach for classification beyond the learned label space in data streams. Proceedings of the 31st Annual ACM Symposium on Applied Computing (SAC ’16), Pisa Italy.10.1145/2851613.2851652 ↗
34. Yalcin, A., Erdem, Z., and Gurgen, F. (2007, January 7–9). Ensemble based incremental SVM classifiers for changing environments. Proceedings of the 2007 22nd International Symposium on Computer and Information Sciences, Ankara, Turkey.10.1109/ISCIS.2007.4456862 ↗
35. Altendeitering, M., and Dübler, S. (2021, January 15–18). Scalable Detection of Concept Drift: A Learning Technique Based on Support Vector Machines. Proceedings of the 30th International Conference on Flexible Automation and Intelligent Manufacturing (FAIM2021), Athens, Greece.
36. Cauwenberghs, G., and Poggio, T. (2000, January 1). Incremental and decremental support vector machine learning. Proceedings of the 13th International Conference on Neural Information Processing Systems (NIPS’00), Denver, CO, USA.
37. Diehl, C.P., and Cauwenberghs, G. (2003, January 20–24). SVM incremental learning, adaptation and optimization. Proceedings of the International Joint Conference on Neural Networks, Portland, OR, USA.
38. Incremental support vector learning: Analysis, implementation and applications
39. Gâlmeanu, H., and Andonie, R. (2008, January 3–6). Implementation issues of an incremental and decremental SVM. Proceedings of the 18th International Conference on Artificial Neural Networks (ICANN ’08), Prague, Czech Republic.
40. Gâlmeanu, H., and Andonie, R. (2009, January 24–26). A multi-class incremental and decremental SVM approach using adaptive directed acyclic graphs. Proceedings of the 2009 International Conference on Adaptive and Intelligent Systems, Klagenfurt, Austria.10.1109/ICAIS.2009.27 ↗
41. Elomaa, T., Mannila, H., and Toivonen, H. (2002). On-line support vector machine regression. Machine Learning: ECML 2002, Springer.10.1007/3-540-36755-1 ↗
42. Accurate on-line support vector regression10.1162/089976603322385117 ↗
43. Incremental and decremental SVM for regression10.15837/ijccc.2016.6.2744 ↗
44. Incremental cost-sensitive Support Vector Machine with linear-exponential loss10.1109/ACCESS.2020.3015954 ↗
45. Pesaranghader, A. (2021, October 03). The Tornado Framework for Data Stream Mining (Python Implementation). Available online: https://github.com/alipsgh/tornado.
46. Mahdi, O.A., Pardede, E., Ali, N., and Cao, J. (2020). Fast Reaction to Sudden Concept Drift in the Absence of Class Labels. Appl. Sci., 10.10.3390/app10020606 ↗
47. Huang, D.T.J., Koh, Y.S., Dobbie, G., and Bifet, A. (2015). Drift detection using stream volatility. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer.10.1007/978-3-319-23528-8_26 ↗
48. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables10.1016/S0168-1699(99)00046-0 ↗
49. Centre for Open Software Innovation, The University of Waikato (2020, May 01). Datasets—MOA. Available online: https://moa.cms.waikato.ac.nz/datasets/.
50. Bifet, A., and Kirkby, R. (2009). Data Stream Mining: A Practical Approach, MOA, The University of Waikato, Centre for Open Software Innovation. Available online: https://www.cs.waikato.ac.nz/~abifet/MOA/StreamMining.pdf.