journal article·2024-11-27Open access

Negative Selection Algorithm for Unsupervised Anomaly Detection

Michał Bereta

Applied Sciences· Volume 14· Issue 23

Abstract

In this work, we present a modification of the well-known Negative Selection Algorithm (NSA), inspired by the process of T-cell generation in the immune system. The approach employs spherical detectors and was initially developed in the context of semi-supervised anomaly detection. The novelty of this work lies in proposing an adapted version of the NSA for unsupervised anomaly detection. The goal is to develop a method that can be applied to datasets that may not only represent self-data but also contain a small percentage of anomalies, which must be detected without prior knowledge of their locations. The proposed unsupervised algorithm leverages neighborhood sampling and ensemble methods to enhance its performance. We conducted comparative tests with 11 other algorithms across 17 datasets with varying characteristics. The results demonstrate that the proposed algorithm is competitive. The proposed algorithm performs well across multiple metrics, including accuracy, AUC, precision, recall, F1 score, Cohen’s kappa, and Matthews correlation coefficient. It consistently ranks among the top algorithms for recall, indicating its effectiveness in scenarios where detecting all existing anomalies is critical, even at the expense of some increase in false positives. Further research is possible and may focus on exploring normalization procedures, improving threshold automation, and extending the method for more detailed anomaly confidence assessments.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Artificial Immune Systems Applications · Anomaly Detection Techniques and Applications · Network Security and Intrusion Detection

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Computer Science, Cracow University of Technology, 31-155 Kraków, Poland

References (46)

Show source-provided references
1. Mehrotra, K.G., Mohan, C.K., Huang, H., Mehrotra, K.G., Mohan, C.K., and Huang, H. (2017). Anomaly Detection Principles and Algorithms, Springer International Publishing. [1st ed.].10.1007/978-3-319-67526-8 ↗
2. Lu, T., Wang, L., and Zhao, X. (2023). Review of Anomaly Detection Algorithms for Data Streams. Appl. Sci., 13.10.3390/app13106353 ↗
3. A comprehensive survey of anomaly detection techniques for high dimensional big data10.1186/s40537-020-00320-x ↗
4. Ali, A., Abd Razak, S., Othman, S.H., Eisa, T.A.E., Al-Dhaqm, A., Nasser, M., Elhassan, T., Elshafie, H., and Saif, A. (2022). Financial Fraud Detection Based on Machine Learning: A Systematic Literature Review. Appl. Sci., 12.10.3390/app12199637 ↗
5. Intelligent financial fraud detection: A comprehensive review10.1016/j.cose.2015.09.005 ↗
6. Prasad, S.K., Routray, S., Khurana, R., and Sahni, S. (2009, January 12–13). A Taxonomy of Frauds and Fraud Detection Techniques. Proceedings of the Information Systems, Technology and Management, Ghaziabad, India.10.1007/978-3-642-00405-6 ↗
7. Current trends in AI and ML for cybersecurity: A state-of-the-art survey10.1080/23311916.2023.2272358 ↗
8. Data-Driven Network Anomaly Detection with Cyber Attack and Defense Visualization10.3390/jcp4020012 ↗
9. Anomalous distributed traffic: Detecting cyber security attacks amongst microservices using graph convolutional networks10.1016/j.cose.2022.102728 ↗
10. Collaborative intrusion detection using weighted ensemble averaging deep neural network for coordinated attack detection in heterogeneous network10.1007/s10207-024-00891-3 ↗
11. Ensemble of Autoencoders for Anomaly Detection in Biomedical Data: A Narrative Review10.1109/ACCESS.2024.3360691 ↗
12. Zhang, C., Xiao, X., and Wu, C. (2020). Medical Fraud and Abuse Detection System Based on Machine Learning. Int. J. Environ. Res. Public Health, 17.10.3390/ijerph17197265 ↗
13. Outlier-based detection of unusual patient-management actions: An ICU study10.1016/j.jbi.2016.10.002 ↗
14. Big data analytics for preventive medicine10.1007/s00521-019-04095-y ↗
15. Review of fault detection techniques for predictive maintenance10.1108/JQME-10-2020-0107 ↗
16. Abdallah, M., Joung, B.G., Lee, W.J., Mousoulis, C., Raghunathan, N., Shakouri, A., Sutherland, J.W., and Bagchi, S. (2023). Anomaly Detection and Inter-Sensor Transfer Learning on Smart Manufacturing Datasets. Sensors, 23.10.3390/s23010486 ↗
17. Detecting anomalies in time series data from a manufacturing system using recurrent neural networks10.1016/j.jmsy.2020.12.007 ↗
18. Enhanced Anomaly Detection in Manufacturing Processes Through Hybrid Deep Learning Techniques10.1109/ACCESS.2023.3308698 ↗
19. A comprehensive review on insider trading detection using artificial intelligence10.1007/s42001-024-00284-5 ↗
20. Woo, K.Y., Mai, C., McAleer, M., and Wong, W.K. (2020). Review on Efficiency and Anomalies in Stock Markets. Economies, 8.10.3390/economies8010020 ↗
21. A machine learning attack on illegal trading10.1016/j.jbankfin.2022.106735 ↗
22. Anomaly Detection in Connected and Autonomous Vehicles: A Survey, Analysis, and Research Challenges10.1109/ACCESS.2024.3361829 ↗
23. Kim, T.H., Ojo, S., Krichen, M., and Alamro, M.A. (2024). Single and Mixed Sensory Anomaly Detection in Connected and Automated Vehicle Sensor Networks. Electronics, 13.10.3390/electronics13101885 ↗
24. Predictive maintenance enabled by machine learning: Use cases and challenges in the automotive industry10.1016/j.ress.2021.107864 ↗
25. Anomaly detection: A survey10.1145/1541880.1541882 ↗
26. Parametric Methods for Anomaly Detection in Aggregate Traffic10.1109/TNET.2010.2070845 ↗
27. Adaptive kernel density-based anomaly detection for nonlinear systems10.1016/j.knosys.2017.10.009 ↗
28. One Class Process Anomaly Detection Using Kernel Density Estimation Methods10.1109/TSM.2022.3181468 ↗
29. Proximity-based density description with regularized reconstruction algorithm for anomaly detection10.1016/j.ins.2023.119816 ↗
30. Detecting outliers from pairwise proximities: Proximity isolation forests10.1016/j.patcog.2023.109334 ↗
31. Huang, J.W., Zhong, M.X., and Jaysawal, B.P. (2020). TADILOF: Time Aware Density-Based Incremental Local Outlier Detection in Data Streams. Sensors, 20.10.3390/s20205829 ↗
32. A hybrid unsupervised clustering-based anomaly detection method10.26599/TST.2019.9010051 ↗
33. Supervised Anomaly Detection in Uncertain Pseudoperiodic Data Streams10.1145/2806890 ↗
34. Multiclass Anomaly Detection in Flight Data Using Semi-Supervised Explainable Deep Learning Model
35. Latent feature reconstruction for unsupervised anomaly detection10.1007/s10489-023-04767-2 ↗
36. Low-rank reconstruction-based autoencoder for robust fault detection10.1016/j.conengprac.2022.105156 ↗
37. Deep Isolation Forest for Anomaly Detection10.1109/TKDE.2023.3270293 ↗
38. Forrest, S., Perelson, A., Allen, L., and Cherukuri, R. (1994, January 16–18). Self-nonself discrimination in a computer. Proceedings of the 1994 IEEE Computer Society Symposium on Research in Security and Privacy, Oakland, CA, USA.
39. Dasgupta, D. (1999). An Overview of Artificial Immune Systems and Their Applications. Artificial Immune Systems and Their Applications, Springer.10.1007/978-3-642-59901-9 ↗
40. Anomaly Detection Using Real-Valued Negative Selection10.1023/A:1026195112518 ↗
41. V-detector: An efficient negative selection algorithm with “probably adequate” detector coverage10.1016/j.ins.2008.12.015 ↗
42. Application of the feature-detection rule to the Negative Selection Algorithm10.1016/j.eswa.2012.12.016 ↗
43. Negative Selection Algorithm: Recent Improvements and Its Application in Intrusion Detection System
44. Anomaly process detection using negative selection algorithm and classification techniques10.1007/s12530-019-09317-1 ↗
45. A Novel Fault Diagnosis Method Based on Improved Negative Selection Algorithm
46. Ali, M. (2024, September 10). PyCaret: An Open Source, Low-Code Machine Learning Library in Python; PyCaret version 1.0.0. Available online: https://pycaret.org/.