Time Series Anomaly Detection for KPIs Based on Correlation Analysis and HMM
Source abstract
Abstract
KPIs (Key Performance Indicators) in distributed systems may involve a variety of anomalies, which will lead to system failure and huge losses. Detecting KPI anomalies in the system is very important. This paper presents a time series anomaly detection method based on correlation analysis and HMM. Correlation analysis is used to obtain the correlation between abnormal KPIs in the system, thereby reducing the false alarm rate of anomaly detection. The HMM (Hidden Markov Model) is used for anomaly detection by finding the close relationship between abnormal KPIs. In our correlation analysis of abnormal KPIs, firstly, the time series prediction model (1D-CNN-TCN) is proposed. The residual sequence is obtained by calculating the residual between the predicted value and the actual value. The residual sequence can highlight the abnormal segment in each data point and improve the accuracy of anomaly screening. According to the obtained residual sequence, these abnormal KPIs are preliminarily screened out from the historical data. Next, KPI correlation analysis is performed, and the correlation score is obtained by adding a sliding window onto the obtained anomaly index residual sequence. The correlation analysis based on the residual sequence can eliminate the interference of the original data fluctuation itself. Then, a correlation matrix of abnormal KPIs is constructed using the obtained correlation scores. In anomaly detection, the constructed correlation matrix is processed to obtain the adaptive parameters of the HMM model, and the trained HMM is used to quickly discover the abnormal KPI that may cause a KPI anomaly. Experiments on public data sets show that the method obtains good results.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Network Security and Intrusion Detection · Anomaly Detection Techniques and Applications · Software System Performance and Reliability
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China
Navigation College, Dalian Maritime University, Dalian 116026, China
Funding
the National Key R&D Program of China · the LiaoNing Revitalization Talents Program
References (36)
Show source-provided references
1. Intelligent operation and maintenance based on machine learning
2. Su, Y., Zhao, Y., Xia, W., Liu, R., Bu, J., Zhu, J., Cao, Y., Li, H., Niu, C., and Zhang, Y. (2019, January 24–25). CoFlux: Robustly correlating KPIs by fluctuations for service troubleshooting. Proceedings of the International Symposium on Quality of Service (IWQoS), Phoenix, AZ, USA.10.1145/3326285.3329048 ↗
3. Laptev, N., Amizadeh, S., and Flint, I. (2015, January 10–13). Generic and Scalable Framework for Automated Time-series Anomaly Detection. Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Sydney, Australia.10.1145/2783258.2788611 ↗
4. Liu, D.P., Zhao, Y.J., Xu, H.W., Sun, Y.Q., Pei, D., Luo, J., Jing, X.W., and Feng, M. (2015, January 28–30). Opprentice: Towards Practical and Automatic Anomaly Detection Through Machine Learning. Proceedings of the ACM Internet Measurement Conference(IMC), Tokyo, Japan.10.1145/2815675.2815679 ↗
5. Niu, Z., Yu, K., and Wu, X. (2020). LSTM-Based VAE-GAN for Time-Series Anomaly Detection. Sensors, 20.10.3390/s20133738 ↗
6. AURORA: A Unified fRamework fOR Anomaly detection on multivariate time series10.1007/s10618-021-00771-7 ↗
7. Xu, H., Chen, W., Zhao, N., Li, Z., Bu, J., Li, Z., Liu, Y., Zhao, Y., Pei, D., and Feng, Y. (2018, January 23–27). Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications. Proceedings of the 2018 World Wide Web Conference (WWW), Lyon, France.10.1145/3178876.3185996 ↗
8. Li, T., Comer, M.L., Delp, E.J., Desai, S.R., Mathieson, J.L., Foster, R.H., and Chan, M.W. (2020, January 7–14). Anomaly Scoring for Prediction-Based Anomaly Detection in Time Series. Proceedings of the 2020 IEEE Aerospace Conference (AeroConf), Yellowstone Conference Center, Big Sky, MT, USA.10.1109/AERO47225.2020.9172442 ↗
9. Approach of measuring and predicting software system state based on hidden Markov model
10. Zhou, Z., Zhang, Y., and Wang, S. (2021). A Coordination System between Decision Making and Controlling for Autonomous Collision Avoidance of Large Intelligent Ships. J. Mar. Sci. Eng., 9.10.3390/jmse9111202 ↗
11. Jiang, J.R., Kao, J.B., and Li, Y.L. (2021). Semi-Supervised Time Series Anomaly Detection Based on Statistics and Deep Learning. Appl. Sci, 11.10.3390/app11156698 ↗
12. Yang, X., Latecki, L.J., and Pokrajac, D. (May, January 30). Outlier Detection with Globally Optimal Exemplar-Based GMM. Proceedings of the International Conference on Data Mining (SDM), Sparks, NV, USA.
13. Li, Z., Zhao, Y., Liu, R., and Pei, D. (2018, January 4–6). Robust and Rapid Clustering of KPIs for Large-Scale Anomaly Detection. Proceedings of the 26th IEEE/ACM International Symposium on Quality of Service (IWQoS), Banff, AB, Canada.10.1109/IWQoS.2018.8624168 ↗
14. Qu, Z., Lun, S., Wang, X., Zheng, S., and Song, X. (2018, January 15–17). A Unsupervised Learning Method of Anomaly Detection Using GRU. Proceedings of the IEEE International Conference on Big Data and Smart Computing (BigComp), Shanghai, China.10.1109/BigComp.2018.00126 ↗
15. Provotar, O.I., Linder, Y.M., and Veres, M.M. (2019, January 18–20). Unsupervised Anomaly Detection in Time Series Using LSTM-Based Autoencoders. Proceedings of the IEEE International Conference on Advanced Trends in Information Theory (ATIT), Kyiv, Ukraine.10.1109/ATIT49449.2019.9030505 ↗
16. Hundman, K., Constantinou, V., Laporte, C., Colwell, I., and Soderstrom, T. (2018, January 19–23). Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding. Proceedings of the 24th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), London, UK.10.1145/3219819.3219845 ↗
17. Bai, S., Kolter, J.Z., and Koltun, V. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv.
18. Convolutional neural network-based data anomaly detection method using multiple information for structural health monitoring10.1002/stc.2296 ↗
19. Ren, H.S., Xu, B.X., Wang, Y.J., Yi, C., Huang, C.R., Kou, X.Y., Xing, T., Yang, M., Tong, J., and Zhang, Q. (2019, January 4–8). Time-Series Anomaly Detection Service at Microsoft. Proceedings of the 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Anchorage, AK, USA.10.1145/3292500.3330680 ↗
20. Unified model for collective and point anomaly detection using stacked temporal convolution networks
21. Weng, J., Wang, J.H., Yang, J., and Yang, Y. (2017, January 14–16). Root cause analysis of anomalies of multitier services in public clouds. Proceedings of the 25th IEEE/ACM International Symposium on Quality of Service (IWQoS), Vilanova, Spain.
22. Kao, J.B., and Jiang, J.R. (2019, January 3–6). Anomaly Detection for Univariate Time Series with Statistics and Deep Learning. Proceedings of the IEEE Eurasia Conference on IOT, Communication and Engineering (IEEE ECICE), Yunlin, Taiwan.10.1109/ECICE47484.2019.8942727 ↗
23. Luo, C., Lou, J., Lin, Q., Fu, Q., Ding, R., Zhang, D., and Wang, Z. (, January 24–27August). Correlating events with time series for incident diagnosis. Proceedings of the 20th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), New York, NY, USA.
24. Deng, S., Zhang, N., Zhang, W., Chen, J., Pan, J.Z., and Chen, H. (2019, January 13–17). Knowledge-Driven Stock Trend Prediction and Explanation via Temporal Convolutional Network. Proceedings of the World Wide Web Conference (WWW), San Francisco, CA, USA.10.1145/3308560.3317701 ↗
25. Li, J., Wu, B., Sun, X., and Wang, Y. (2021, January 19–25). Causal Hidden Markov Model for Time Series Disease Forecasting. Proceedings of the Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA. Virtual.10.1109/CVPR46437.2021.01193 ↗
26. The three sigma rule10.1080/00031305.1994.10476030 ↗
27. Backpropagation Applied to Handwritten Zip Code Recognition10.1162/neco.1989.1.4.541 ↗
28. Yu, F., and Koltun, V. (2016, January 2–4). Multi-Scale Context Aggregation by Dilated Convolutions. Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico.
29. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
30. A linear memory algorithm for baum
31. A tutorial on hidden Markov models and selected applications in speech recognition10.1109/5.18626 ↗
32. (2021, October 08). 2021 International AIOps Challenge. Available online: http://iops.ai/competition_detail/?competition_id=17&flag=1.
33. Application of LSTM for short term fog forecasting based on meteorological elements10.1016/j.neucom.2019.12.129 ↗
34. A CNN-LSTM-Based Model to Forecast Stock Prices10.1155/2020/6622927 ↗
35. Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is All you Need. Proceedings of the Neural Information Processing Systems (NIPS), Long Beach, CA, USA.
36. (2021, October 08). 2018 International AIOps Challenge. Available online: http://iops.ai/competition_detail/?competition_id=5&flag=1.