Machine Learning and Deep Learning Methods for Intrusion Detection Systems: A Survey
Source abstract
Abstract
Networks play important roles in modern life, and cyber security has become a vital research area. An intrusion detection system (IDS) which is an important cyber security technique, monitors the state of software and hardware running in the network. Despite decades of development, existing IDSs still face challenges in improving the detection accuracy, reducing the false alarm rate and detecting unknown attacks. To solve the above problems, many researchers have focused on developing IDSs that capitalize on machine learning methods. Machine learning methods can automatically discover the essential differences between normal data and abnormal data with high accuracy. In addition, machine learning methods have strong generalizability, so they are also able to detect unknown attacks. Deep learning is a branch of machine learning, whose performance is remarkable and has become a research hotspot. This survey proposes a taxonomy of IDS that takes data objects as the main dimension to classify and summarize machine learning-based and deep learning-based IDS literature. We believe that this type of taxonomy framework is fit for cyber security researchers. The survey first clarifies the concept and taxonomy of IDSs. Then, the machine learning algorithms frequently used in IDSs, metrics, and benchmark datasets are introduced. Next, combined with the representative literature, we take the proposed taxonomic system as a baseline and explain how to solve key IDS issues with machine learning and deep learning techniques. Finally, challenges and future developments are discussed by reviewing recent representative studies.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Network Security and Intrusion Detection · Anomaly Detection Techniques and Applications · Advanced Malware Detection Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
State Key Laboratory of Software Development Environment, Beihang University, Beijing 100191, China
References (93)
Show source-provided references
1. Anderson, J.P. (1980). Computer Security Threat Monitoring and Surveillance, James P. Anderson Company. Technical Report.
2. Michie, D., Spiegelhalter, D.J., and Taylor, C. (1994). Machine Learning, Neurall and Statistical Classification, Ellis Horwood Series in Artificial Intelligence.
3. A survey of data mining and machine learning methods for cyber security intrusion detection10.1109/COMST.2015.2494502 ↗
4. Machine learning and deep learning methods for cybersecurity10.1109/ACCESS.2018.2836950 ↗
5. Survey on anomaly detection using data mining techniques10.1016/j.procs.2015.08.220 ↗
6. Denning, D.E. (1987). An intrusion-detection model. IEEE Trans. Softw. Eng., 222–232.10.1109/TSE.1987.232894 ↗
7. Heberlein, L.T., Dias, G.V., Levitt, K.N., Mukherjee, B., Wood, J., and Wolber, D. (1990, January 7–9). A network security monitor. Proceedings of the 1990 IEEE Computer Society Symposium on Research in Security and Privacy, Oakland, CA, USA.10.1109/RISP.1990.63859 ↗
8. A novel SVM by combining kernel principal component analysis and improved chaotic particle swarm optimization for intrusion detection10.1007/s00500-014-1332-7 ↗
9. Syarif, A.R., and Gata, W. (2017, January 31). Intrusion detection system using hybrid binary PSO and K-nearest neighborhood algorithm. Proceedings of the 2017 11th International Conference on Information & Communication Technology and System (ICTS), Surabaya, Indonesia.10.1109/ICTS.2017.8265667 ↗
10. Two-tier network anomaly detection model: A machine learning approach10.1007/s10844-015-0388-x ↗
11. Network Intrusion Detection System (NIDS) in Cloud Environment based on Hidden Naïve Bayes Multiclass Classifier10.23851/mjs.v28i2.508 ↗
12. Shah, R., Qian, Y., Kumar, D., Ali, M., and Alvi, M. (2017). Network intrusion detection through discriminative feature selection by using sparse logistic regression. Future Internet, 9.10.3390/fi9040081 ↗
13. Clustering approach based on mini batch kmeans for intrusion detection system over big data10.1109/ACCESS.2018.2810267 ↗
14. Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.A. (2008, January 5–9). Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland.10.1145/1390156.1390294 ↗
15. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion
16. Deng, J., Zhang, Z., Marchi, E., and Schuller, B. (2013, January 2–5). Sparse autoencoder-based feature transfer learning for speech emotion recognition. Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, Geneva, Switzerland.10.1109/ACII.2013.90 ↗
17. Hinton, G.E. (2012). A practical guide to training restricted Boltzmann machines. Neural Networks: Tricks of the Trade, Springer.10.1007/978-3-642-35289-8_32 ↗
18. A fast learning algorithm for deep belief nets10.1162/neco.2006.18.7.1527 ↗
19. Boureau, Y.l., Cun, Y.L., and Ranzato, M.A. (2008, January 8–10). Sparse feature learning for deep belief networks. Proceedings of the 21st Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada.
20. Zhao, G., Zhang, C., and Zheng, L. (2017, January 21–24). Intrusion detection using deep belief network and probabilistic neural network. Proceedings of the 2017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC), Guangzhou, China.10.1109/CSE-EUC.2017.119 ↗
21. Alrawashdeh, K., and Purdy, C. (2016, January 18–20). Toward an online anomaly intrusion detection system based on deep learning. Proceedings of the 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), Anaheim, CA, USA.10.1109/ICMLA.2016.0040 ↗
22. Yang, Y., Zheng, K., Wu, C., Niu, X., and Yang, Y. (2019). Building an effective intrusion detection system using the modified density peak clustering algorithm and deep belief networks. Appl. Sci., 9.10.3390/app9020238 ↗
23. Sharif Razavian, A., Azizpour, H., Sullivan, J., and Carlsson, S. (2014, January 23–28). CNN features off-the-shelf: An astounding baseline for recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Columbus, OH, USA.10.1109/CVPRW.2014.131 ↗
24. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). Imagenet classification with deep convolutional neural networks. Proceedings of the 26th Annual Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
25. Face recognition: A convolutional neural-network approach10.1109/72.554195 ↗
26. Graves, A., Mohamed, A.R., and Hinton, G. (2013, January 26–31). Speech recognition with deep recurrent neural networks. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6638947 ↗
27. Graves, A., and Jaitly, N. (2014, January 21–26). Towards end-to-end speech recognition with recurrent neural networks. Proceedings of the International Conference on Machine Learning, Beijing, China.
28. Sutskever, I., Vinyals, O., and Le, Q.V. (2014, January 8–13). Sequence to sequence learning with neural networks. Proceedings of the Annual Conference on Neural Information Processing Systems 2014, Montreal, QC, Canada.
29. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
30. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv.
31. Bidirectional recurrent neural networks10.1109/78.650093 ↗
32. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). Why should i trust you?: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939778 ↗
33. Lundberg, S.M., and Lee, S.I. (2017, January 4–9). A unified approach to interpreting model predictions. Proceedings of the Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA.
34. Li, J., Monroe, W., and Jurafsky, D. (2016). Understanding neural networks through representation erasure. arXiv.
35. Fong, R.C., and Vedaldi, A. (2017, January 22–29). Interpretable explanations of black boxes by meaningful perturbation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.371 ↗
36. (2019, October 16). DARPA1998 Dataset. Available online: http://www.ll.mit.edu/r-d/datasets/1998-darpa-intrusion-detection-evaluation-dataset.
37. (2019, October 16). KDD99 Dataset. Available online: http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.
38. (2019, October 16). NSL-KDD99 Dataset. Available online: https://www.unb.ca/cic/datasets/nsl.html.
39. Moustafa, N., and Slay, J. (2015, January 10–12). UNSW-NB15: A comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). Proceedings of the 2015 Military Communications And Information Systems Conference (MilCIS), Canberra, Australia.10.1109/MilCIS.2015.7348942 ↗
40. Mayhew, M., Atighetchi, M., Adler, A., and Greenstadt, R. (2015, January 10–12). Use of machine learning in big data analytics for insider threat detection. Proceedings of the MILCOM 2015—2015 IEEE Military Communications Conference, Canberra, Australia.10.1109/MILCOM.2015.7357562 ↗
41. Hu, L., Li, T., Xie, N., and Hu, J. (2015, January 15–17). False positive elimination in intrusion detection based on clustering. Proceedings of the 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD), Zhangjiajie, China.
42. CNN and RNN based payload classification methods for attack detection10.1016/j.knosys.2018.08.036 ↗
43. TR-IDS: Anomaly-based intrusion detection through text-convolutional neural network and random forest10.1155/2018/4943509 ↗
44. Deep-Full-Range: A Deep Learning Based Network Encrypted Traffic Classification and Intrusion Detection Framework10.1109/ACCESS.2019.2908225 ↗
45. Network intrusion detection through stacking dilated convolutional autoencoders10.1155/2017/4184196 ↗
46. Rigaki, M., and Garcia, S. (2018, January 24). Bringing a gan to a knife-fight: Adapting malware communication to avoid detection. Proceedings of the 2018 IEEE Security and Privacy Workshops (SPW), San Francisco, CA, USA.10.1109/SPW.2018.00019 ↗
47. Goeschel, K. (April, January 30). Reducing false positives in intrusion detection systems using data-mining techniques utilizing support vector machines, decision trees, and naive Bayes for off-line analysis. Proceedings of the SoutheastCon 2016, Norfolk, VA, USA.10.1109/SECON.2016.7506774 ↗
48. Parallel KNN and Neighborhood Classification Implementations on GPU for Network Intrusion Detection
49. Potluri, S., Ahmed, S., and Diedrich, C. (2018). Convolutional Neural Networks for Multi-class Intrusion Detection System. Mining Intelligence and Knowledge Exploration, Springer.10.1007/978-3-030-05918-7_20 ↗
50. Zhang, B., Yu, Y., and Li, J. (2018, January 20–24). Network Intrusion Detection Based on Stacked Sparse Autoencoder and Binary Tree Ensemble Method. Proceedings of the 2018 IEEE International Conference on Communications Workshops (ICC Workshops), Kansas City, MO, USA.10.1109/ICCW.2018.8403759 ↗
51. Zhang, H., Yu, X., Ren, P., Luo, C., and Min, G. (2019). Deep Adversarial Learning in Intrusion Detection: A Data Augmentation Enhanced Framework. arXiv.
52. SVM-DT-based adaptive and collaborative intrusion detection10.1109/JAS.2017.7510730 ↗
53. Ma, T., Wang, F., Cheng, J., Yu, Y., and Chen, X. (2016). A hybrid spectral clustering and deep neural network ensemble algorithm for intrusion detection in sensor networks. Sensors, 16.10.3390/s16101701 ↗
54. Ahmim, A., Maglaras, L., Ferrag, M.A., Derdour, M., and Janicke, H. (2019, January 29–31). A novel hierarchical intrusion detection system based on decision tree and rules-based models. Proceedings of the 2019 15th International Conference on Distributed Computing in Sensor Systems (DCOSS), Santorini Island, Greece.10.1109/DCOSS.2019.00059 ↗
55. Alseiari, F.A.A., and Aung, Z. (2015, January 20–23). Real-time anomaly-based distributed intrusion detection systems for advanced Metering Infrastructure utilizing stream data mining. Proceedings of the 2015 International Conference on Smart Grid and Clean Energy Technologies (ICSGCE), Offenburg, Germany.10.1109/ICSGCE.2015.7454287 ↗
56. Yuan, X., Li, C., and Li, X. (2017, January 29–31). DeepDefense: identifying DDoS attack via deep learning. Proceedings of the 2017 IEEE International Conference on Smart Computing (SMARTCOMP), Hong Kong, China.10.1109/SMARTCOMP.2017.7946998 ↗
57. Radford, B.J., Apolonio, L.M., Trias, A.J., and Simpson, J.A. (2018). Network traffic anomaly detection using recurrent neural networks. arXiv.
58. HAST-IDS: Learning hierarchical spatial-temporal features using deep neural networks to improve intrusion detection10.1109/ACCESS.2017.2780250 ↗
59. Design of intelligent KNN-based alarm filter using knowledge-based alert verification in intrusion detection10.1002/sec.1307 ↗
60. McElwee, S., Heaton, J., Fraley, J., and Cannady, J. (2017, January 23–25). Deep learning for prioritizing and responding to intrusion detection alerts. Proceedings of the MILCOM 2017—2017 IEEE Military Communications Conference (MILCOM), Baltimore, MD, USA.10.1109/MILCOM.2017.8170757 ↗
61. Tran, N.N., Sarker, R., and Hu, J. (2017, January 23–25). An Approach for Host-Based Intrusion Detection System Design Using Convolutional Neural Network. Proceedings of the International Conference on Mobile Networks and Management, Chiba, Japan.
62. Tuor, A., Kaplan, S., Hutchinson, B., Nichols, N., and Robinson, S. (2017, January 4–9). Deep learning for unsupervised insider threat detection in structured cybersecurity data streams. Proceedings of the Workshops at the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.
63. Bohara, A., Thakore, U., and Sanders, W.H. (2016, January 19–21). Intrusion detection in enterprise systems by combining and clustering diverse monitor data. Proceedings of the Symposium and Bootcamp on the Science of Security, Pittsburgh, PA, USA.10.1145/2898375.2898400 ↗
64. Uwagbole, S.O., Buchanan, W.J., and Fan, L. (2017, January 8–12). Applied machine learning predictive analytics to SQL injection attack detection and prevention. Proceedings of the 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), Lisbon, Portugal.10.23919/INM.2017.7987433 ↗
65. Vartouni, A.M., Kashi, S.S., and Teshnehlab, M. (March, January 28). An anomaly detection method to detect web attacks using Stacked Auto-Encoder. Proceedings of the 2018 6th Iranian Joint Congress on Fuzzy and Intelligent Systems (CFIS), Kerman, Iran.10.1109/CFIS.2018.8336654 ↗
66. Potluri, S., and Diedrich, C. (2016, January 6–9). Accelerated deep neural networks for enhanced Intrusion Detection System. Proceedings of the 2016 IEEE 21st International Conference on Emerging Technologies and Factory Automation (ETFA), Berlin, Germany.10.1109/ETFA.2016.7733515 ↗
67. Pektaş, A., and Acarman, T. (2018). Deep learning to detect botnet via network flow summaries. Neural Comput. Appl., 1–13.10.1007/s00521-018-3595-x ↗
68. Kim, J., Shin, N., Jo, S.Y., and Kim, S.H. (2017, January 13–16). Method of intrusion detection using deep neural network. Proceedings of the 2017 IEEE International Conference on Big Data and Smart Computing (BigComp), Jeju, Korea.
69. A deep learning approach to network intrusion detection10.1109/TETCI.2017.2772792 ↗
70. A decision tree classifier for intrusion detection priority tagging10.4236/jcc.2015.34006 ↗
71. Effective intrusion detection system using data mining technique
72. Improving intrusion detection system based on snort rules for network probe attacks detection with association rules technique of data mining10.5614/itbj.ict.res.appl.2015.8.3.4 ↗
73. Performance comparison of intrusion detection systems and application of machine learning to Snort system10.1016/j.future.2017.10.016 ↗
74. Fouladi, R.F., Kayatas, C.E., and Anarim, E. (2016, January 27–29). Frequency based DDoS attack detection approach using naive Bayes classification. Proceedings of the 2016 39th International Conference on Telecommunications and Signal Processing (TSP), Vienna, Austria.10.1109/TSP.2016.7760838 ↗
75. Detecting distributed denial of service attacks using data mining techniques
76. Niyaz, Q., Sun, W., and Javaid, A.Y. (2016). A deep learning based DDoS detection system in software-defined networking (SDN). arXiv.10.4108/eai.28-12-2017.153515 ↗
77. Yadav, S., and Subramanian, S. (2016, January 11–13). Detection of Application Layer DDoS attack by feature learning using Stacked AutoEncoder. Proceedings of the 2016 International Conference on Computational Techniques in Information and Communication Technologies (ICCTICT), New Delhi, India.10.1109/ICCTICT.2016.7514608 ↗
78. Nguyen, S.N., Nguyen, V.Q., Choi, J., and Kim, K. (2018, January 2–4). Design and implementation of intrusion detection system using convolutional neural network for dos detection. Proceedings of the 2nd International Conference on Machine Learning and Soft Computing, Phu Quoc Island, Vietnam.10.1145/3184066.3184089 ↗
79. Bontemps, L., McDermott, J., and Le-Khac, N.A. (2016, January 23–25). Collective anomaly detection based on long short-term memory recurrent neural networks. Proceedings of the International Conference on Future Data and Security Engineering, Tho City, Vietnam.10.1007/978-3-319-48057-2_9 ↗
80. Bapat, R., Mandya, A., Liu, X., Abraham, B., Brown, D.E., Kang, H., and Veeraraghavan, M. (2018, January 27). Identifying malicious botnet traffic using logistic regression. Proceedings of the 2018 Systems and Information Engineering Design Symposium (SIEDS), Charlottesville, VA, USA.10.1109/SIEDS.2018.8374749 ↗
81. Abdelhamid, N., Thabtah, F., and Abdel-jaber, H. (2017, January 22–24). Phishing detection: A recent intelligent machine learning comparison based on models content and features. Proceedings of the 2017 IEEE International Conference on Intelligence and Security Informatics (ISI), Beijing, China.10.1109/ISI.2017.8004877 ↗
82. Intrusion detection system based on decision tree over big data in fog environment10.1155/2018/4680867 ↗
83. He, Z., Zhang, T., and Lee, R.B. (2017, January 26–28). Machine learning based DDoS attack detection from source side in cloud. Proceedings of the 2017 IEEE 4th International Conference on Cyber Security and Cloud Computing (CSCloud), New York, NY, USA.10.1109/CSCloud.2017.58 ↗
84. Doshi, R., Apthorpe, N., and Feamster, N. (2018, January 24). Machine learning ddos detection for consumer internet of things devices. Proceedings of the 2018 IEEE Security and Privacy Workshops (SPW), San Francisco, CA, USA.10.1109/SPW.2018.00013 ↗
85. N-BaIoT— Network-based detection of IoT botnet attacks using deep autoencoders10.1109/MPRV.2018.03367731 ↗
86. Leveraging LSTM networks for attack detection in fog-to-things communications10.1109/MCOM.2018.1701270 ↗
87. Detection of false data injection attacks against state estimation in smart grids based on a mixture Gaussian distribution learning method10.1049/iet-cps.2017.0013 ↗
88. Real-time detection of false data injection attacks in smart grid: A deep learning-based intelligent mechanism10.1109/TSG.2017.2703842 ↗
89. An Innovative Two-Stage Fuzzy kNN-DST Classifier for Unknown Intrusion Detection
90. Random forest modeling for network intrusion detection system10.1016/j.procs.2016.06.047 ↗
91. Feature selection based hybrid anomaly intrusion detection system using K means and RBF kernel function10.1016/j.procs.2015.03.174 ↗
92. Jabbar, M., Aluvalu, R., and Reddy, S. (2017, January 24–26). Cluster based ensemble classification for intrusion detection system. Proceedings of the 9th International Conference on Machine Learning and Computing, Singapore.10.1145/3055635.3056595 ↗
93. Guo, W., Mu, D., Xu, J., Su, P., Wang, G., and Xing, X. (2018, January 15). Lemna: Explaining deep learning based security applications. Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, Toronto, ON, Canada.10.1145/3243734.3243792 ↗