Review of Anomaly Detection Algorithms for Data Streams
Source abstract
Abstract
With the rapid development of emerging technologies such as self-media, the Internet of Things, and cloud computing, massive data applications are crossing the threshold of the era of real-time analysis and value realization, which makes data streams ubiquitous in all kinds of industries. Therefore, detecting anomalies in such data streams could be very important and full of challenges. For example, in industries such as electricity and finance, data stream anomalies often contain information that can help avoiding risks and support decision making. However, most traditional anomaly detection algorithms rely on acquiring global information about the data, which is hard to apply to stream data scenarios. Currently, the reviews of the algorithm in the field of anomaly detection, both domestically and internationally, tend to focus on the exposition of anomaly detection algorithms in static data environments, while lacking in the induction and analysis of anomaly detection algorithms in the context of streaming data. As a result, unlike the existing literature reviews, this review provides the current mainstream anomaly detection algorithms in data streaming scenarios and categorizes them into three types on the basis of their fundamental principles: (1) based on offline learning; (2) based on semi-online learning; (3) based on online learning. This review discusses the current state of research on data stream anomaly detection and studies the key issues in various algorithms for detecting anomalies in data streams on the basis of concise summarization. Moreover, the review conducts a detailed comparison of the pros and cons of the algorithms. Finally, the future challenges in the field are analyzed, and future research directions are proposed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Anomaly Detection Techniques and Applications · Data Stream Mining Techniques · Network Security and Intrusion Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Management, Beijing Information Science and Technology University, Beijing 100192, China
Funding
The National Key Research and Development Program of China
References (64)
Show source-provided references
1. Korycki, Ł., Cano, A., and Krawczyk, B. (2019, January 9–13). Active Learning with Abstaining Classifiers for Imbalanced Drifting Data Streams. Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA.10.1109/BigData47090.2019.9006453 ↗
2. Bhatia, S., Jain, A., Li, P., Kumar, R., and Hooi, B. (2021, January 19–23). MSTREAM: Fast Anomaly Detection in Multi-Aspect Streams. Proceedings of the Web Conference, Ljubljana, Slovenia.10.1145/3442381.3450023 ↗
3. Data stream clustering: A review10.1007/s10462-020-09874-x ↗
4. Data stream analysis: Foundations, major tasks and tools10.1002/widm.1405 ↗
5. Real-time big data processing for anomaly detection: A survey10.1016/j.ijinfomgt.2018.08.006 ↗
6. Artificial intelligence based anomaly detection of energy consumption in buildings: A review, current trends and new perspectives10.1016/j.apenergy.2021.116601 ↗
7. Detecting regions of maximal divergence for spatio-temporal anomaly detection10.1109/TPAMI.2018.2823766 ↗
8. Random partitioning forest for point-wise and collective anomaly detection—Application to network intrusion detection10.1109/TIFS.2021.3050605 ↗
9. Data stream classification with novel class detection: A review, comparison and challenges10.1007/s10115-021-01582-4 ↗
10. Progress in outlier detection techniques: A survey10.1109/ACCESS.2019.2932769 ↗
11. Eltanbouly, S., Bashendy, M., AlNaimi, N., Chkirbene, Z., and Erbad, A. (2020, January 2–5). Machine Learning Techniques for Network Anomaly Detection: A Survey. Proceedings of the 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), Dawḥah, Qatar.10.1109/ICIoT48696.2020.9089465 ↗
12. Anomaly detection methods for categorical data: A review10.1145/3312739 ↗
13. Machine learning for anomaly detection: A systematic review10.1109/ACCESS.2021.3083060 ↗
14. Deep learning for anomaly detection: A review10.1145/3439950 ↗
15. A review on outlier/anomaly detection in time series data
16. Anomaly detection for IoT time-series data: A survey10.1109/JIOT.2019.2958185 ↗
17. A survey on anomaly detection in evolving data10.1145/3229329.3229332 ↗
18. A survey of outlier detection in high dimensional data streams10.1016/j.cosrev.2022.100463 ↗
19. Algorithm AS 136: A k-means clustering algorithm10.2307/2346830 ↗
20. Nearest neighbor pattern classification10.1109/TIT.1967.1053964 ↗
21. LOF: Identifying density-based local outliers10.1145/335191.335388 ↗
22. Support vector data description10.1023/B:MACH.0000008084.60811.49 ↗
23. Liu, F.T., Ting, K.M., and Zhou, Z.H. (2008, January 15–19). Isolation Forest. Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, Pisa, Italy.10.1109/ICDM.2008.17 ↗
24. Sathe, S., and Aggarwal, C.C. (2016, January 12–16). Subspace Outlier Detection in Linear Time with Randomized Hashing. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.10.1109/ICDM.2016.0057 ↗
25. DeepAnT: A deep learning approach for unsupervised anomaly detection in time series10.1109/ACCESS.2018.2886457 ↗
26. Anomaly detection of time series with smoothness-inducing sequential variational auto-encoder10.1109/TNNLS.2020.2980749 ↗
27. Akcay, S., Atapour-Abarghouei, A., and Breckon, T.P. (2018, January 2–6). Ganomaly: Semi-Supervised Anomaly Detection via Adversarial Training. Proceedings of the Computer Vision–ACCV 2018, 14th Asian Conference on Computer Vision, Perth, Australia. Revised Selected Papers, Part III 14.10.1007/978-3-030-20893-6_39 ↗
28. Transfer Learning Auto-Encoder Neural Networks for Anomaly Detection of DDoS Generating IoT Devices10.1155/2022/8221351 ↗
29. Fast memory efficient local outlier detection in data streams10.1109/TKDE.2016.2597833 ↗
30. NETS: Extremely fast outlier detection from a data stream via set-based processing10.14778/3342263.3342269 ↗
31. Amer, M., Goldstein, M., and Abdennadher, S. (2013, January 11). Enhancing One-Class Support Vector Machines for Unsupervised Anomaly Detection. Proceedings of the ACM SIGKDD Workshop on Outlier Detection and Description, Chicago, IL, USA.10.1145/2500853.2500857 ↗
32. Angiulli, F., and Fassetti, F. (2007, January 6–10). Detecting distance-based outliers in streams of data. Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management, Lisbon, Portugal.10.1145/1321440.1321552 ↗
33. Distance-based outlier queries in data streams: The novel task and algorithms10.1007/s10618-009-0159-9 ↗
34. Kontaki, M., Gounaris, A., Papadopoulos, N., Tsichlas, K., and Manolopoulos, Y. (2011, January 11–16). Continuous Monitoring of Distance-Based Outliers over Data Streams. Proceedings of the IEEE 27th International Conference on Data Engineering, Hannover, Germany.10.1109/ICDE.2011.5767923 ↗
35. Effificient and flflexible algorithms for monitoring distance-based outliers over data streams10.1016/j.is.2015.07.006 ↗
36. Continuous outlier mining of streaming data in flink10.1016/j.is.2020.101569 ↗
37. Na, G.S., Kim, D., and Yu, H. (2018, January 19–23). Dilof: Effective and Memory Efficient Local Outlier Detection in Data Streams. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3220022 ↗
38. Borah, A., Gruenwald, L., Leal, E., and Panjei, E. (2021, January 15–18). A GPU Algorithm for Detecting Contextual Outliers in Multiple Concurrent Data Streams. Proceedings of the 2021 IEEE International Conference on Big Data (Big Data), Virtual Conference.10.1109/BigData52589.2021.9671460 ↗
39. An anomaly detection approach based on isolation forest algorithm for streaming data using sliding window10.3182/20130902-3-CN-3020.00044 ↗
40. Sun, H., He, Q., Liao, K., Sellis, T., Guo, L., Zhang, X., Shen, J., and Chen, F. (2019, January 9–13). Fast Anomaly Detection in Multiple Multi-Dimensional Data Streams. Proceedings of the 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA.10.1109/BigData47090.2019.9006354 ↗
41. Anomaly detection via online oversampling principal component analysis10.1109/TKDE.2012.99 ↗
42. Novoa-Paradela, D., Fontenla-Romero, O., and Guijarro-Berdiñas, B. (2020, January 19–24). Online Learning for Anomaly Detection via Subdivisible Convex Hulls. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.10.1109/IJCNN48605.2020.9207036 ↗
43. Streaming anomaly detection using randomized matrix sketching10.14778/2850583.2850593 ↗
44. A framework of virtual war room and matrix sketch-based streaming anomaly detection for microservice systems10.1109/ACCESS.2020.2977464 ↗
45. Hulten, G., Spencer, L., and Domingos, P. (2001, January 26–29). Mining Time-Changing Data Streams. Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/502512.502529 ↗
46. Bifet Figuerol, A.C., and Gavaldà Mestre, R. (2009). Adaptive Parameter-Free Learning from Evolving Data Streams, UPC Universitat Politècnica de Catalunya.10.1007/978-3-642-03915-7_22 ↗
47. Manapragada, C., Webb, G.I., and Salehi, M. (2018, January 19–23). Extremely Fast Decision Tree. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3220005 ↗
48. Garcia-Martin, E., Bifet, A., Lavesson, N., König, R., and Linusson, H. (2022). Green Accelerated Hoeffding Tree. arXiv.
49. Sahoo, D., Pham, Q., Lu, J., and Hoi, S.C. (2018, January 13–19). Online Deep Learning: Learning Deep Neural Networks on the Fly. Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden.10.24963/ijcai.2018/369 ↗
50. Yuan, Y., Adhatarao, S.S., Lin, M., Yuan, Y., Liu, Z., and Fu, X. (2020, January 6–9). Ada: Adaptive Deep Log Anomaly Detector. Proceedings of the IEEE INFOCOM 2020—IEEE Conference on Computer Communications, Virtual.10.1109/INFOCOM41043.2020.9155487 ↗
51. Zong, B., Song, Q., Min, M.R., Cheng, W., Lumezanu, C., Cho, D., and Chen, H. (May, January 30). Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection. Proceedings of the International Conference on Learning Representations, Vancouver, VA, Canada.
52. Yoon, S., Lee, Y., Lee, J.G., and Lee, B.S. (2022, January 14–18). Adaptive Model Pooling for Online Deep Anomaly Detection from a Complex Evolving Data Stream. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA.10.1145/3534678.3539348 ↗
53. S-DPs: An SDN-based DDoS protection system for smart grids10.1155/2021/6629098 ↗
54. Concept drift detection in data stream mining: A literature review
55. Tellis, V.M., and D’Souza, D.J. (2018, January 23–24). Detecting Anomalies in Data Stream Using Efficient Techniques: A Review. Proceedings of the 2018 International Conference on Control, Power, Communication and Computing Technologies (ICCPCCT), Kannur, India.10.1109/ICCPCCT.2018.8574310 ↗
56. Reducing the dimensionality of data with neural networks10.1126/science.1127647 ↗
57. Zhang, X., Dou, W., He, Q., Zhou, R., Leckie, C., Kotagiri, R., and Salcic, Z. (2017, January 9–12). LSHiForest: A Generic Framework for Fast Tree Isolation Based Ensemble Anomaly Analysis. Proceedings of the 2017 IEEE 33rd International Conference on Data Engineering (ICDE), Los Angeles, CA, USA.10.1109/ICDE.2017.145 ↗
58. Liberty, E. (2013, January 11–14). Simple and Deterministic Matrix Sketching. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA.10.1145/2487575.2487623 ↗
59. Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions
60. Sahoo, D., Pham, Q., Lu, J., and Hoi, S.C. (2017). Online deep learning: Learning deep neural networks on the fly. arXiv.10.24963/ijcai.2018/369 ↗
61. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv.10.3115/v1/D14-1179 ↗
62. Alrawashdeh, K., and Purdy, C. (2018, January 23–25). Fast Hardware Assisted Online Learning Using Unsupervised Deep Learning Structure for Anomaly Detection. Proceedings of the 2018 International Conference on Information and Computer Technologies (ICICT), Dekalb, IL, USA.10.1109/INFOCT.2018.8356855 ↗
63. FLAGS: A methodology for adaptive anomaly detection and root cause analysis on sensor data streams by fusing expert knowledge with machine learning10.1016/j.future.2020.10.015 ↗
64. Hierarchical routing protocols for wireless sensor networks: Functional and performance analysis10.1155/2021/7459368 ↗