Towards a Reliable Comparison and Evaluation of Network Intrusion Detection Systems Based on Machine Learning Approaches
Source abstract
Abstract
Presently, we are living in a hyper-connected world where millions of heterogeneous devices are continuously sharing information in different application contexts for wellness, improving communications, digital businesses, etc. However, the bigger the number of devices and connections are, the higher the risk of security threats in this scenario. To counteract against malicious behaviours and preserve essential security services, Network Intrusion Detection Systems (NIDSs) are the most widely used defence line in communications networks. Nevertheless, there is no standard methodology to evaluate and fairly compare NIDSs. Most of the proposals elude mentioning crucial steps regarding NIDSs validation that make their comparison hard or even impossible. This work firstly includes a comprehensive study of recent NIDSs based on machine learning approaches, concluding that almost all of them do not accomplish with what authors of this paper consider mandatory steps for a reliable comparison and evaluation of NIDSs. Secondly, a structured methodology is proposed and assessed on the UGR’16 dataset to test its suitability for addressing network attack detection problems. The guideline and steps recommended will definitively help the research community to fairly assess NIDSs, although the definitive framework is not a trivial task and, therefore, some extra effort should still be made to improve its understandability and usability further.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Network Security and Intrusion Detection · Internet Traffic Analysis and Secure E-voting · Advanced Malware Detection Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science & Engineering, School of Engineering, University of Cádiz, 11519 Cádiz, Spain
Department of Computer Engineering, School of Engineering, University of Burgos, 09006 Burgos, Spain
Department of Automatic, Electronic, Computer Architecture & Communication Networks Engineering, School of Engineering, University of Cádiz, 11519 Cádiz, Spain
Funding
Ministerio de Ciencia, Innovación y Universidades
References (44)
Show source-provided references
1. ENISA (2019, September 22). ENISA Threat Landscape Report 2017, Available online: https://www.enisa.europa.eu/publications/enisa-threat-landscape-report-2017.
2. Network Intrusion Detection for IoT Security Based on Learning Techniques10.1109/COMST.2019.2896380 ↗
3. ENISA (2019, September 22). ENISA Threat Landscape Report 2018, Available online: https://www.enisa.europa.eu/publications/enisa-threat-landscape-report-2018.
4. Di Pietro, R., and Mancini, L.V. (2008). Intrusion Detection Systems, Springer.
5. Anomaly-based network intrusion detection: Techniques, systems and challenges10.1016/j.cose.2008.08.003 ↗
6. University of California (2019, September 20). KDD Cup 1999 Dataset. Available online: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.
7. Tavallaee, M., Bagheri, E., Lu, W., and Ghorbani, A.A. (2009, January 8–10). A detailed analysis of the KDD CUP 99 data set. Proceedings of the 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications, Ottawa, ON, Canada.10.1109/CISDA.2009.5356528 ↗
8. Moustafa, N., and Slay, J. (2015, January 10–12). UNSW-NB15: A comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). Proceedings of the 2015 Military Communications and Information Systems Conference (MilCIS), Canberra, Australia.10.1109/MilCIS.2015.7348942 ↗
9. The Evaluation of Network Anomaly Detection Systems: Statistical Analysis of the UNSW-NB15 Data Set and the Comparison with the KDD99 Data Set10.1080/19393555.2015.1125974 ↗
10. Network Anomaly Detection: Methods, Systems and Tools10.1109/SURV.2013.052213.00046 ↗
11. Liu, H., and Lang, B. (2019). Machine Learning and Deep Learning Methods for Intrusion Detection Systems: A Survey. Appl. Sci., 9.10.3390/app9204396 ↗
12. Semi-supervised Multivariate Statistical Network Monitoring for Learning Security Threats10.1109/TIFS.2019.2894358 ↗
13. KDD Cup 99 Data Sets: A Perspective on the Role of Data Sets in Network Intrusion Detection Research10.1109/MC.2018.2888764 ↗
14. UGR‘16: A new dataset for the evaluation of cyclostationarity-based network IDSs10.1016/j.cose.2017.11.004 ↗
15. Real time intrusion detection system for ultra-high-speed big data environments10.1007/s11227-015-1615-5 ↗
16. Generating realistic intrusion detection system dataset based on fuzzy qualitative modeling10.1016/j.jnca.2017.03.018 ↗
17. Sharafaldin, I., Lashkari, A.H., and Ghorbani, A.A. (2018, January 22–24). Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization. Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP), Funchal, Portugal.10.5220/0006639801080116 ↗
18. Li, Z., Rios, A.L.G., Xu, G., and Trajković, L. (2019, January 26–29). Machine Learning Techniques for Classifying Network Anomalies and Intrusions. Proceedings of the 2019 IEEE International Symposium on Circuits and Systems (ISCAS), Sapporo, Japan.10.1109/ISCAS.2019.8702583 ↗
19. Le, T.T.H., Kim, Y., and Kim, H. (2019). Network Intrusion Detection Based on Novel Feature Selection Model and Various Recurrent Neural Networks. Appl. Sci., 9.10.3390/app9071392 ↗
20. Cordero, C.G., Vasilomanolakis, E., Wainakh, A., Mühlhäuser, M., and Nadjm-Tehrani, S. (2019). On generating network traffic datasets with synthetic attacks for intrusion detection. arXiv.
21. A novel statistical technique for intrusion detection systems10.1016/j.future.2017.01.029 ↗
22. A hybrid intrusion detection system based on ABC-AFS algorithm for misuse and anomaly detection10.1016/j.comnet.2018.02.028 ↗
23. Divekar, A., Parekh, M., Savla, V., Mishra, R., and Shirole, M. (2018, January 25–27). Benchmarking datasets for Anomaly-based Network Intrusion Detection: KDD CUP 99 alternatives. Proceedings of the 2018 IEEE 3rd International Conference on Computing, Communication and Security (ICCCS), Kathmandu, Nepal.10.1109/CCCS.2018.8586840 ↗
24. Performance evaluation of intrusion detection based on machine learning using Apache Spark10.1016/j.procs.2018.01.091 ↗
25. Feature Analysis, Evaluation and Comparisons of Classification Algorithms Based on Noisy Intrusion Dataset10.1016/j.procs.2016.07.345 ↗
26. An empirical comparison of botnet detection methods10.1016/j.cose.2014.05.011 ↗
27. Zhang, J., Liang, Q., Jiang, R., and Li, X. (2019). A Feature Analysis Based Identifying Scheme Using GBDT for DDoS with Multiple Attack Vectors. Appl. Sci., 9.10.3390/app9214633 ↗
28. García Cordero, C., Hauke, S., Mühlhäuser, M., and Fischer, M. (2016, January 12–14). Analyzing flow-based anomaly intrusion detection using Replicator Neural Networks. Proceedings of the 2016 14th Annual Conference on Privacy, Security and Trust (PST), Auckland, New Zealand.10.1109/PST.2016.7906980 ↗
29. PCA-based multivariate statistical network monitoring for anomaly detection10.1016/j.cose.2016.02.008 ↗
30. Layered Approach Using Conditional Random Fields for Intrusion Detection10.1109/TDSC.2008.20 ↗
31. A survey of network-based intrusion detection data sets10.1016/j.cose.2019.06.005 ↗
32. Camacho, J., Maciá-Fernández, G., Díaz-Verdejo, J., and García-Teodoro, P. (May, January 27). Tackling the Big Data 4 vs for anomaly detection. Proceedings of the 2014 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), Toronto, ON, Canada.10.1109/INFCOMW.2014.6849282 ↗
33. Multivariate Big Data Analysis for intrusion detection: 5 steps from the haystack to the needle10.1016/j.cose.2019.101603 ↗
34. Pérez-Villegas, A., García-Jiménez, J., and Camacho, J. (2019, December 20). FaaC (Feature-as-a-Counter) Parser—Github. Available online: https://github.com/josecamachop/FCParser.
35. Regularization Paths for Generalized Linear Models via Coordinate Descent10.18637/jss.v033.i01 ↗
36. Freeman, D., and Chio, C. (2018). Machine Learning and Security, O’Reilly Media.
37. Random Search for Hyper-Parameter Optimization
38. Magán Carrión, R., and Diaz-Cano, I. (2019, January 5–7). Evaluación de Algoritmos de Clasificación para la Detección de Ataques en Red Sobre Conjuntos de Datos Reales: UGR’16 Dataset como caso de Estudio. Proceedings of the V Jornadas Nacionales de Investigación en Ciberseguridad, Cáceres, Spain.
39. Snoek, J., Larochelle, H., and Adams, R.P. (2012). Practical Bayesian Optimization of Machine Learning Algorithms. Advances in Neural Information Processing Systems 25, Curran Associates, Inc.
40. Bishop, C. (2006). Pattern Recognition and Machine Learning, Springer Inc.. Information Science and Statistics.
41. Random Forests10.1023/A:1010933404324 ↗
42. Data Mining Techniques in Intrusion Detection Systems: A Systematic Literature Review10.1109/ACCESS.2018.2872784 ↗
43. Free Framework for Machine Learning (2019, December 26). Cámara UI-3250CP-C-HQ. Available online: https://github.com/ucadatalab/ff4ml.
44. A Study on the use of Non-parametric Tests for Analyzing the Evolutionary Algorithms’ Behaviour: A Case Study on the CEC’05 Special Session on Real Parameter Optimization10.1007/s10732-008-9080-4 ↗