Selecting the Suitable Resampling Strategy for Imbalanced Data Classification Regarding Dataset Properties. An Approach Based on Association Models
Source abstract
Abstract
In many application domains such as medicine, information retrieval, cybersecurity, social media, etc., datasets used for inducing classification models often have an unequal distribution of the instances of each class. This situation, known as imbalanced data classification, causes low predictive performance for the minority class examples. Thus, the prediction model is unreliable although the overall model accuracy can be acceptable. Oversampling and undersampling techniques are well-known strategies to deal with this problem by balancing the number of examples of each class. However, their effectiveness depends on several factors mainly related to data intrinsic characteristics, such as imbalance ratio, dataset size and dimensionality, overlapping between classes or borderline examples. In this work, the impact of these factors is analyzed through a comprehensive comparative study involving 40 datasets from different application areas. The objective is to obtain models for automatic selection of the best resampling strategy for any dataset based on its characteristics. These models allow us to check several factors simultaneously considering a wide range of values since they are induced from very varied datasets that cover a broad spectrum of conditions. This differs from most studies that focus on the individual analysis of the characteristics or cover a small range of values. In addition, the study encompasses both basic and advanced resampling strategies that are evaluated by means of eight different performance metrics, including new measures specifically designed for imbalanced data classification. The general nature of the proposal allows the choice of the most appropriate method regardless of the domain, avoiding the search for special purpose techniques that could be valid for the target data.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Imbalanced Data Classification Techniques · Text and Document Classification Technologies · Machine Learning and Data Classification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Data Mining Research Group, University of Salamanca, 37008 Salamanca, Spain
Funding
Consejería de Educación, Junta de Castilla y León
References (82)
Show source-provided references
1. Learning from imbalanced data10.1109/TKDE.2008.239 ↗
2. A novel intelligent classification model for breast cancer diagnosis10.1016/j.ipm.2018.10.014 ↗
3. Random forest based ensemble classifiers for predicting healthcare-associated infections in Intensive Care Units
4. Sánchez-Hernández, F., Ballesteros-Herráez, J.C., Kraiem, M.S., Sánchez-Barba, M., and Moreno- García, M.N. (2019). Predictive Modeling of ICU Healthcare-Associated Infections from Imbalanced Data. Using Ensembles and a Clustering-Based Undersampling Approach. Appl. Sci., 9.10.3390/app9245287 ↗
5. Success/failure prediction of noninvasive mechanical ventilation in intensive care units. Using multiclassifiers and feature selection methods10.3414/ME14-01-0015 ↗
6. Machine Learning Methods for Mortality Prediction of Polytraumatized Patients in Intensive Care Units. Dealing with Imbalanced and High-Dimensional Data
7. Multiclassifier systems for predicting neurological outcome of patients with severe trauma and polytrauma in intensive care units10.1007/s10916-017-0789-1 ↗
8. Cognitive-inspired domain adaptation of sentiment lexicons10.1016/j.ipm.2018.11.002 ↗
9. Rodríguez-González, A., Tuñas, J.M., Prieto Santamaría, L., Fernández Peces-Barba, D., Menasalvas Ruiz, E., Jaramillo, A., Cotarelo, M., Conejo Fernández, A.J., Arce, A., and Gil, A. (2020). Identifying Polarity in Tweets from an Imbalanced Dataset about Diseases and Vaccines Using a Meta-Model Based on Machine Learning Techniques. Appl. Sci., 10.10.3390/app10249019 ↗
10. Web spam classification method based on deep belief networks10.1016/j.eswa.2017.12.016 ↗
11. Fraud detection for E-commerce transactions by employing a prudential Multiple Consensus model
12. A study on software fault prediction techniques10.1007/s10462-017-9563-5 ↗
13. Shilling attacks against collaborative recommender systems: A review10.1007/s10462-018-9655-x ↗
14. An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics10.1016/j.ins.2013.07.007 ↗
15. Ranawana, R., and Palade, V. (2006, January 16–21). Optimized precision—A new measure for classifier performance evaluation. In Proceeding of the IEEE Congress on Computational Intelligence, Vancouver, BC, Canada.
16. Index of balanced accuracy: A performance measure for skewed class distributions
17. Moreno, M.N., Segrera, S., and López, V. (2005). Association Rules: Problems, Solutions and New Applications, TAMIDA.
18. Sattar, A., and Kang, B.H. (2006). z-SVM: An SVM for Improved Classification of Imbalanced Data. AI 2006: Advances in Artificial Intelligence, Springer.
19. SVMs modeling for highly imbalanced classification10.1109/TSMCB.2008.2002909 ↗
20. Visa, S., and Ralescu, A. (2003, January 21). Learning imbalanced and overlapping classes using fuzzy sets. Proceedings of the ICML 2003, Workshop: Learning with Imbalanced Data Sets II, Washington, DC, USA.
21. Hierarchical fuzzy rule based classification systems with genetic rule selection for imbalanced data-sets10.1016/j.ijar.2008.11.004 ↗
22. Classification with class Imbalance problem: A review
23. A fuzzy twin support vector machine based on information entropy for class imbalance learning10.1007/s00521-018-3551-9 ↗
24. Intuitionistic fuzzy twin support vector machines10.1109/TFUZZ.2019.2893863 ↗
25. Extreme rebalancing for SVM: A case Study10.1145/1007730.1007739 ↗
26. Combating the small sample Using Feature selection10.1109/TKDE.2009.187 ↗
27. García, V., Sánchez, J.S., Mollineda, R.A., Alejo, R., and Sotoca, J.M. (2007, January 11−14). The class Imbalance problem in pattern Classification and Learning. Proceedings of the II Congreso Español de Informática, Zaragoza, Spain.
28. Juszczak, P., and Duin, R.P.W. (2003, January 21). Uncertainty sampling methods for one class classifers. Proceedings of the International Conference on Machine Learning, Workshop on Learning with Imbalance Datasets II, Washington, DC, USA.
29. Liu, X.Y., and Zhou, Z. (2006, January 18–22). The Influence of Class Imbalance on Cost-Sensitive Learning: An Empirical Study. Proceedings of the Sixth International Conference on Data Mining, Hong Kong, China.10.1109/ICDM.2006.158 ↗
30. Ying, P.Y. (2009). Learning Pattern Classification Tasks with Imbalanced Datasets. Pattern Recognition, Intech Open.
31. Novel classification method for sensitive problems and uneven datasets based on neural networks and fuzzy logic10.1016/j.asoc.2010.09.001 ↗
32. Lin, H.T. (2010). Cost-sensitive classification: Status and beyond. Workshop on Machine Learning Research in Taiwan: Challenges and Directions, National Taiwan University.
33. Classification of imbalanced data: A review10.1142/S0218001409007326 ↗
34. Analysis of preprocessing vs. cost-sensitive learning for imbalanced classification. Open problems on intrinsic data characteristics10.1016/j.eswa.2011.12.043 ↗
35. SMOTE: Synthetic minority over sampling technique10.1613/jair.953 ↗
36. A generalization of the K-NN rule10.1109/TSMC.1976.5409182 ↗
37. A fuzzy association rule-based classifier for imbalanced classification problems10.1016/j.ins.2021.07.019 ↗
38. Class imbalance learning using fuzzy ART and intuitionistic fuzzy twin support vector machines10.1016/j.ins.2021.07.010 ↗
39. Hulse., J.V., Khoshgoftaa., T.M., and Naplolitano., A. (2007, January 20−24). Experimental perspectives on learning from imbalanced data. Proceedings of the 24th International Conference on Machine Learning, Corvallis, OR, USA.
40. The class imbalance problem: A systematic study10.3233/IDA-2002-6504 ↗
41. More, A. (2016). Survey of resampling techniques for improving classification performance in imbalanced datasets. arXiv.
42. Diversity techniques improve the performance of the best imbalance learning ensembles10.1016/j.ins.2015.07.025 ↗
43. An Ensemble of Heterogeneous Incremental Classifiers for Assisted Reproductive Technology Outcome Prediction10.1109/TCSS.2020.3032640 ↗
44. Study of the impact of resampling methods for contrast pattern based classifiers in imbalanced databases10.1016/j.neucom.2015.04.120 ↗
45. Combining integrated sampling with SVM ensembles for learning from imbalanced datasets10.1016/j.ipm.2010.11.007 ↗
46. A random forests classifier for class imbalanced data10.1016/j.patcog.2019.01.036 ↗
47. Efective data generation for imbalanced learning using conditional generative adversarial networks10.1016/j.eswa.2017.09.030 ↗
48. Dirvanauskas, D., Maskeliunas, R., Raudonis, V., Damaševicius, R., and Scherer, R. (2019). HEMIGEN: Human Embryo Image Generator Based on Generative Adversarial Networks. Sensors, 19.10.3390/s19163578 ↗
49. Shaikh, S., Daudpota, S.M., Imran, A.S., and Kastrati, Z. (2021). Towards Improved Classification Accuracy on Highly Imbalanced Text Dataset Using Deep Neural Language Models. Appl. Sci., 11.10.3390/app11020869 ↗
50. Deep and machine learning approaches for anomaly-based intrusion detection of imbalanced network traffic10.1109/LSENS.2018.2879990 ↗
51. Intrusion Detection of Imbalanced Network Traffic Based on Machine Learning and Deep Learning10.1109/ACCESS.2020.3048198 ↗
52. Preprocessing unbalance data using support vector machine10.1016/j.dss.2012.01.016 ↗
53. A sampling based sentiment mining approach for e-commerce applications10.1016/j.ipm.2016.08.003 ↗
54. Enhancement of conformational B-cell epitope prediction using CluSMOTE10.7717/peerj-cs.275 ↗
55. Data augmentation-based conditional Wasserstein generative adversarial network-gradient penalty for XSS attack detection system10.7717/peerj-cs.328 ↗
56. SMOTE-IBF: Addressing the noisy and borderline examples problem in imbalanced classification by resampling method with filtering10.1016/j.ins.2014.08.051 ↗
57. A method for resampling imbalanced datasets in binary classification tasks for real world problems10.1016/j.neucom.2013.05.059 ↗
58. Rendón, E., Alejo, R., Castorena, C., Isidro-Ortega, F.J., and Granda-Gutiérrez, E.E. (2020). Data Sampling Methods to Deal With the Big Data Multi-Class Imbalance Problem. Appl. Sci., 10.10.3390/app10041276 ↗
59. Interval–valued fuzzy and intuitionistic fuzzy–KNN for imbalanced data classification10.1016/j.eswa.2021.115510 ↗
60. Wan, Z., Zhang, Y., and He, H. (December, January 27). Variational autoencoder based synthetic data generation for imbalanced learning. Proceedings of the 2017 IEEE Symposium Series on Computational Intelligence, Honolulu, HI, USA.10.1109/SSCI.2017.8285168 ↗
61. A conditional variational autoencoder based self-transferred algorithm for imbalanced classification10.1016/j.knosys.2021.106756 ↗
62. Multivariate Discretization for Associative Classification in a Sparse Data Application Domain
63. Agrawal, R., and Srikant, R. (1994, January 12−15). Fast Algorithms for mining association rules in large databases. Proceedings of the 20th Int. Conference on Very Large Databases, Santiago, Chile.
64. The lattice-based approaches for mining association rules: A review10.1002/widm.1181 ↗
65. Han, J., Pei, H., and Yin, Y. (2000). Mining Frequent Patterns without Candidate Generation. Proceedings of the Management of Data (SIGMOD’00), ACM Press.10.1145/342009.335372 ↗
66. GPU-based swarm intelligence for Association Rule Mining in big databases10.3233/IDA-173785 ↗
67. Ruiz, M.D., Gómez-Romero, J., Martin Bautista, M.J., Vila, M.A., and Delgado, M. (2015, January 30). Fuzzy meta-association rules. Proceedings of the 2015 Conference of the International Fuzzy Systems Association and the European Society for Fuzzy Logic and Technology (IFSA-EUSFLAT-15), Gijón, Spain.10.2991/ifsa-eusflat-15.2015.37 ↗
68. Djenouri, Y., Belhadi, A., Fournier-Viger, P., and Lin, J.C. (2018). Discovering Strong Meta Association Rules Using Bees Swarm Optimization, Springer. PAKDD (Workshops).10.1007/978-3-030-04503-6_21 ↗
69. The condensed nearest neighbor rule10.1109/TIT.1968.1054155 ↗
70. Asymptotic properties of nearest neighbor rules using edited data10.1109/TSMC.1972.4309137 ↗
71. Experiments on the edited condensed nearest neighbor rule10.1016/0020-0255(78)90040-3 ↗
72. Fisher, D.H. Addressing the curse of imbalanced training sets: One side selection. Proceedings of the Fourteenth International Conference on Machine Learning.
73. Handling Imbalance Datasets: A review
74. Napierala, K., Stefanowski, J., and Wilk, S. (2010, January 28−30). Learning from imbalanced data in presence of noise and Borderline examples. Proceedings of the 7th International Conference on Rough Sets and Current Trends in Computing, Lecture Notes on Artificial Intelligence, Warsaw, Poland.10.1007/978-3-642-13529-3_18 ↗
75. Class imbalances versus small disjuncts10.1145/1007730.1007737 ↗
76. The use of the area under the ROC curve in the evaluation of machine learning algorithms
77. On the Importance of the validation technique for classification with imbalanced Datasets: Addressing covariate shift when data is skewed10.1016/j.ins.2013.09.038 ↗
78. Random Forests10.1023/A:1010933404324 ↗
79. Effectiveness of basic and advanced sampling strategies on the classification of imbalanced data. A comparative study using classical and novel metrics10.1007/978-3-319-59650-1_20 ↗
80. Complexity measures of supervised classification problems10.1109/34.990132 ↗
81. How Complex Is Your Classification Problem? A Survey on Measuring Classification Complexity10.1145/3347711 ↗
82. An extension on statistical comparisons of classifiers over multiple datasets for all pairwise comparisons