Is It Worth It? Comparing Six Deep and Classical Methods for Unsupervised Anomaly Detection in Time Series
Source abstract
Abstract
Detecting anomalies in time series data is important in a variety of fields, including system monitoring, healthcare and cybersecurity. While the abundance of available methods makes it difficult to choose the most appropriate method for a given application, each method has its strengths in detecting certain types of anomalies. In this study, we compare six unsupervised anomaly detection methods of varying complexity to determine whether more complex methods generally perform better and if certain methods are better suited to certain types of anomalies. We evaluated the methods using the UCR anomaly archive, a recent benchmark dataset for anomaly detection. We analyzed the results on a dataset and anomaly-type level after adjusting the necessary hyperparameters for each method. Additionally, we assessed the ability of each method to incorporate prior knowledge about anomalies and examined the differences between point-wise and sequence-wise features. Our experiments show that classical machine learning methods generally outperform deep learning methods across a range of anomaly types.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Anomaly Detection Techniques and Applications · Network Security and Intrusion Detection · Time Series Analysis and Forecasting
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Data Science, German Aerospace Center, 07745 Jena, Germany
Faculty of Mathematics and Computer Science, Friedrich Schiller University, 07743 Jena, Germany
References (49)
Show source-provided references
1. A Unifying Review of Deep and Shallow Anomaly Detection10.1109/JPROC.2021.3052449 ↗
2. Healthcare and anomaly detection: Using machine learning to predict anomalies in heart rate data10.1007/s00146-020-00985-1 ↗
3. Tetko, I.V., Kůrková, V., Karpov, P., and Theis, F. (2019). Artificial Neural Networks and Machine Learning—ICANN 2019: Text and Time Series, Springer International Publishing.10.1007/978-3-030-30487-4 ↗
4. A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection10.1109/COMST.2015.2494502 ↗
5. A Multi-Step Anomaly Detection Strategy Based on Robust Distances for the Steel Industry10.1109/ACCESS.2021.3070659 ↗
6. A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-Based Variational Autoencoder10.1109/LRA.2018.2801475 ↗
7. Experimental Comparison and Survey of Twelve Time Series Anomaly Detection Algorithms10.1613/jair.1.12698 ↗
8. Laptev, N., Amizadeh, S., and Flint, I. (2015, January 10–13). Generic and Scalable Framework for Automated Time-Series Anomaly Detection. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’15, Sydney, Australia.10.1145/2783258.2788611 ↗
9. Robust Random Cut Forest Based Anomaly Detection on Streams
10. Detecting Regions of Maximal Divergence for Spatio-Temporal Anomaly Detection10.1109/TPAMI.2018.2823766 ↗
11. Nakamura, T., Imamura, M., Mercer, R., and Keogh, E. (2020, January 17–20). MERLIN: Parameter-Free Discovery of Arbitrary Length Anomalies in Massive Time Series Archives. Proceedings of the 2020 IEEE International Conference on Data Mining (ICDM), Sorrento, Italy.10.1109/ICDM50108.2020.00147 ↗
12. Dai, E., and Chen, J. (2022, January 25–29). Graph-Augmented Normalizing Flows for Anomaly Detection of Multiple Time Series. Proceedings of the 10th International Conference on Learning Representations (ICLR), Virtual. Available online: https://openreview.net/forum?id=45L_dgP48Vd.
13. TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data10.14778/3514061.3514067 ↗
14. Wu, R., and Keogh, E. (2021). Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress. IEEE Trans. Knowl. Data Eng., Early Access.10.1109/TKDE.2021.3112126 ↗
15. Outlier detection for temporal data: A survey10.1109/TKDE.2013.184 ↗
16. Goldstein, M., and Uchida, S. (2014, January 16–17). Behavior analysis using unsupervised anomaly detection. Proceedings of the 10th Joint Workshop on Machine Perception and Robotics (MPR 2014), Online.
17. A review on outlier/anomaly detection in time series data
18. Braei, M., and Wagner, S. (2020). Anomaly Detection in Univariate Time-series: A Survey on the State-of-the-Art. arXiv.
19. Chalapathy, R., and Chawla, S. (2019). Deep Learning for Anomaly Detection: A Survey. arXiv.10.1145/3394486.3406704 ↗
20. Deep Learning for Anomaly Detection: A Review10.1145/3439950 ↗
21. Salehi, M., Mirzaei, H., Hendrycks, D., Li, Y., Rohban, M.H., and Sabokrou, M. (2023, January 18). A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges. Available online: https://openreview.net/forum?id=aRtjVZvbpK.
22. Anomalous Example Detection in Deep Learning: A Survey10.1109/ACCESS.2020.3010274 ↗
23. Forecasting at scale10.1080/00031305.2017.1380080 ↗
24. Time series joins, motifs, discords and shapelets: A unifying view that exploits the matrix profile10.1007/s10618-017-0519-9 ↗
25. Xu, H., Chen, W., Zhao, N., Li, Z., Bu, J., Li, Z., Liu, Y., Zhao, Y., Pei, D., and Feng, Y. (2018, January 23–27). Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications. Proceedings of the 2018 World Wide Web Conference, WWW ’18, Lyon, France.10.1145/3178876.3185996 ↗
26. Lavin, A., and Ahmad, S. (2015, January 9–11). Evaluating real-time anomaly detection algorithms–the Numenta anomaly benchmark. Proceedings of the 2015 IEEE 14th international conference on machine learning and applications (ICMLA), Miami, FL, USA.10.1109/ICMLA.2015.141 ↗
27. Estimation of the Youden Index and its associated cutoff point10.1002/bimj.200410135 ↗
28. Graabæk, S.G., Ancker, E.V., Christensen, A.L., and Fugl, A.R. (2022). An Experimental Comparison of Anomaly Detection Methods for Collaborative Robot Manipulators. IEEE Access, Preprint.10.36227/techrxiv.19006643.v2 ↗
29. Isolation-Based Anomaly Detection10.1145/2133360.2133363 ↗
30. Wang, Y., Wang, Z., Xie, Z., Zhao, N., Chen, J., Zhang, W., Sui, K., and Pei, D. (2020, January 3–6). Practical and White-Box Anomaly Detection through Unsupervised and Active Learning. Proceedings of the 2020 29th International Conference on Computer Communications and Networks (ICCCN), Honolulu, HI, USA.10.1109/ICCCN49398.2020.9209704 ↗
31. Statistical Process Control of Multivariate Processes10.1016/S1474-6670(17)48188-2 ↗
32. Yankov, D., Keogh, E., and Rebbapragada, U. (2007, January 28–31). Disk Aware Discord Discovery: Finding Unusual Time Series in Terabyte Sized Datasets. Proceedings of the Seventh IEEE International Conference on Data Mining (ICDM 2007), Omaha, NE, USA.10.1109/ICDM.2007.61 ↗
33. De Paepe, D., Avendano, D.N., and Van Hoecke, S. (2019, January 19–21). Implications of Z-Normalization in the Matrix Profile. Proceedings of the Pattern Recognition Applications and Methods: 8th International Conference, ICPRAM 2019, Prague, Czech Republic. Revised Selected Papers.
34. Rumelhart, D.E., and McClelland, J.L. (1987). Learning Internal Representations by Error Propagation, MIT Press.10.1016/B978-1-4832-1446-7.50035-2 ↗
35. Autoencoders, Unsupervised Learning, and Deep Architectures
36. Bank, D., Koenigstein, N., and Giryes, R. (2020). Autoencoders. arXiv.
37. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017, January 4–9). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
38. Wu, R., and Keogh, E. (2023, January 30). UCR Anomaly Archive. Available online: https://www.cs.ucr.edu/~Eeamonn/time_series_data_2018/UCR_TimeSeriesAnomalyDatasets2021.zip.
39. Laptev, N., Amizadeh, S., and Billawala, Y. (2023, January 18). S5—A Labeled Anomaly Detection Dataset, Version 1.0 (16M). Available online: https://webscope.sandbox.yahoo.com/catalog.php?datatype=s&did=70.
40. Hundman, K., Constantinou, V., Laporte, C., Colwell, I., and Soderstrom, T. (2018, January 19–23). Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.10.1145/3219819.3219845 ↗
41. Comparison of baseline wander removal techniques considering the preservation of ST changes in the ischemic ECG: A simulation study10.1155/2017/9295029 ↗
42. Wu, R., and Keogh, E. (2023, January 18). UCR_AnomalyDataSets.pptx, Supplemental Material to the UCR Anomaly Archive. Available online: https://www.cs.ucr.edu/~Eeamonn/time_series_data_2018/UCR_TimeSeriesAnomalyDatasets2021.zip.
43. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). Advances in Neural Information Processing Systems 32, Curran Associates, Inc.
44. Siffer, A., Fouque, P.A., Termier, A., and Largouet, C. (2017, January 13–17). Anomaly detection in streams with extreme value theory. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada.10.1145/3097983.3098144 ↗
45. Graphan: Graph-based subsequence anomaly detection10.14778/3415478.3415514 ↗
46. Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., and Pei, D. (2019, January 4–8). Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, Anchorage, AK, USA.10.1145/3292500.3330672 ↗
47. Zhao, H., Wang, Y., Duan, J., Huang, C., Cao, D., Tong, Y., Xu, B., Bai, J., Tong, J., and Zhang, Q. (2020, January 17–20). Multivariate Time-Series Anomaly Detection via Graph Attention Network. Proceedings of the 2020 IEEE International Conference on Data Mining (ICDM), Sorrento, Italy.10.1109/ICDM50108.2020.00093 ↗
48. The use of the area under the ROC curve in the evaluation of machine learning algorithms10.1016/S0031-3203(96)00142-2 ↗
49. Lu, Y., Wu, R., Mueen, A., Zuluaga, M.A., and Keogh, E. (2022, January 14–18). Matrix Profile XXIV: Scaling Time Series Anomaly Detection to Trillions of Datapoints and Ultra-fast Arriving Data Streams. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, Washington, DC, USA.10.1145/3534678.3539271 ↗