A Causality Mining and Knowledge Graph Based Method of Root Cause Diagnosis for Performance Anomaly in Cloud Applications
Source abstract
Abstract
With the development of cloud computing technology, the microservice architecture (MSA) has become a prevailing application architecture in cloud-native applications. Many user-oriented services are supported by many microservices, and the dependencies between services are more complicated than those of a traditional monolithic architecture application. In such a situation, if an anomalous change happens in the performance metric of a microservice, it will cause other related services to be downgraded or even to fail, which would probably cause large losses to dependent businesses. Therefore, in the operation and maintenance job of cloud applications, it is critical to mine the causality of the problem and find its root cause as soon as possible. In this paper, we propose an approach for mining causality and diagnosing the root cause that uses knowledge graph technology and a causal search algorithm. We verified the proposed method on a classic cloud-native application and found that the method is effective. After applying our method on most of the services of a cloud-native application, both precision and recall were over 80%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software System Performance and Reliability · Cloud Computing and Resource Management · Software Engineering Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Software Engineering, Tongji University, Shanghai 201804, China
Donald Bren School of Information and Computer Sciences, University of California, Irvine 6210 Donald Bren Hall, Irvine, CA 92697-3425, USA
References (29)
Show source-provided references
1. Kim, M., Roshan, S., and Sam, S. (2013). Root Cause Detection in a Service-Ooriented Architecture, ACM. ACM SIGMETRICS Performance Evaluation Review 41.1.10.1145/2494232.2465753 ↗
2. Thalheim, J., Rodrigues, A., Akkus, I.E., Bhatotia, P., Chen, R., Viswanath, B., Jiao, L., and Fetzer, C. (2017, January 11–15). Sieve: Actionable insights from monitored metrics in distributed systems. Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference, Las Vegas, NV, USA.
3. Root cause analysis of anomalies of multitier services in public clouds10.1109/TNET.2018.2843805 ↗
4. Marwede, N., Rohr, M., van Hoorn, A., and Hasselbring, W. (2009, January 24–27). Automatic failure diagnosis support in distributed large-scale software systems based on timing behavior anomaly correlation. Proceedings of the IEEE 2009 13th European Conference on Software Maintenance and Reengineering, Kaiserslautern, Germany.10.1109/CSMR.2009.15 ↗
5. An anomaly event correlation engine: Identifying root causes, bottlenecks, and black swans in IT environments
6. Zeng, C., Tang, L., Li, T., Shwartz, L., and Grabarnik, G.Y. (2014, January 17–21). Mining temporal lag from fluctuating events for correlation and root cause analysis. Proceedings of the IEEE 10th International Conference on Network and Service Management (CNSM) and Workshop, Rio de Janeiro, Brazil.10.1109/CNSM.2014.7014137 ↗
7. Lin, Q., Zhang, H., Lou, J.-G., Zhang, Y., and Chen, X. (2016, January 14–22). Log clustering based problem identification for online service systems. Proceedings of the 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), Austin, TX, USA.10.1145/2889160.2889232 ↗
8. Jia, T., Chen, P., Yang, L., Meng, F., and Xu, J. (2017, January 25–30). An approach for anomaly diagnosis based on hybrid graph model with logs for distributed services. Proceedings of the 2017 IEEE International Conference on Web Services (ICWS), Honolulu, HI, USA.10.1109/ICWS.2017.12 ↗
9. Xu, J., Chen, P., Yang, L., Meng, F., and Wang, P. (2017, January 4–6). LogDC: Problem diagnosis for declartively-deployed cloud applications with log. Proceedings of the 2017 IEEE 14th International Conference on e-Business Engineering (ICEBE), Shanghai, China.10.1109/ICEBE.2017.52 ↗
10. Xu, X., Zhu, L., Weber, I., Bass, L., and Sun, D. (2014, January 23–26). POD-diagnosis: Error diagnosis of sporadic operations on cloud applications. Proceedings of the 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, Atlanta, GA, USA.10.1109/DSN.2014.94 ↗
11. Jia, T., Yang, L., Chen, P., Li, Y., Meng, F., and Xu, J. (2017, January 25–30). Logsed: Anomaly diagnosis through mining time-weighted control flow graph in logs. Proceedings of the 2017 IEEE 10th International Conference on Cloud Computing (CLOUD), Honolulu, CA, USA.10.1109/CLOUD.2017.64 ↗
12. Mi, H., Wang, H., Cai, H., Zhou, Y., Lyu, M.R., and Chen, Z. (2012, January 16–20). P-tracer: Path-based performance profiling in cloud computing systems. Proceedings of the 2012 IEEE 36th Annual Computer Software and Applications Conference, Izmir, Turkey.10.1109/COMPSAC.2012.69 ↗
13. Chen, M.Y., Kiciman, E., Fratkin, E., Fox, A., and Brewer, E. (2002, January 23–26). Pinpoint: Problem determination in large, dynamic internet services. Proceedings of the Proceedings International Conference on Dependable Systems and Networks, Washington, DC, USA.
14. Gao, H., Yang, Z., Bhimani, J., Wang, T., Wang, J., Sheng, B., and Mi, N. (August, January 31). AutoPath: Harnessing parallel execution paths for efficient resource allocation in multi-stage big data frameworks. Proceedings of the 2017 26th International Conference on Computer Communication and Networks (ICCCN), Vancouver, BC, Canada.10.1109/ICCCN.2017.8038381 ↗
15. CloRExPa: Cloud resilience via execution path analysis10.1016/j.future.2012.05.010 ↗
16. Pengfei, C., Yong, Q., Pengfei, Z., and Di, H. (May, January 27). CauseInfer: Automatic and distributed performance diagnosis with hierarchical causality graph in large distributed systems. Proceedings of the IEEE Conference on Computer Communications, Toronto, ON, Canada.
17. CauseInfer: Automated end-to-end performance diagnosis with hierarchical causality graph in cloud environment
18. Lin, J., Pengfei, C., and Zibin, Z. (2018, January 12–15). Microscope: Pinpoint performance issues with causal graphs in micro-service environments. Proceedings of the International Conference on Service-Oriented Computing, Hangzhou, China.10.1007/978-3-030-03596-9_1 ↗
19. Ping, W., Jingmin, X., Meng, M., Weilan, L., Disheng, P., Yuan, W., and Pengfei, C. (2018, January 1–4). Cloudranger: Root cause identification for cloud native systems. Proceedings of the 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), Washington, DC, USA.
20. Nie, X., Zhao, Y., Sui, K., Pei, D., Chen, Y., Qu, X., Zhao, Y., Sui, K., Pei, D., and Chen, Y. (2016, January 9–11). Mining causality graph for automatic web-based service diagnosis. Proceedings of the 2016 IEEE 35th International Performance Computing and Communications Conference (IPCCC), Las Vegas, NV, USA.10.1109/PCCC.2016.7820614 ↗
21. Lin, W., Ma, M., Pan, D., and Wang, P. (2018, January 17–19). FacGraph: Frequent anomaly correlation graph mining for root cause diagnose in MSA. Proceedings of the 2018 IEEE 37th International Performance Computing and Communications Conference (IPCCC), Orlando, FL, USA.10.1109/PCCC.2018.8711092 ↗
22. Network application profiling with traffic causality graphs10.1002/nem.1865 ↗
23. Combining knowledge modeling and machine learning for alarm root cause analysis10.3182/20130619-3-RU-3018.00057 ↗
24. Qiu, J., Qingfeng, D., and Chongshu, Q. (2019). KPI-TSAD: A Time-Series Anomaly Detector for KPI Monitoring in Cloud Applications. Symmetry, 11.10.3390/sym11111350 ↗
25. An algorithm for fast recovery of sparse causal graphs10.1177/089443939100900106 ↗
26. Heim, P., Hellmann, S., Lehmann, J., Lohmann, S., and Stegemann, T. (2009, January 2–4). RelFinder: Revealing relationships in RDF knowledge bases. Proceedings of the International Conference on Semantic and Digital Media Technologies, Graz, Austria.10.1007/978-3-642-10543-2_21 ↗
27. Causality: Models, reasoning, and inference, by judea pearl, cambridge university press, 200010.1017/S0266466603004109 ↗
28. Luo, C., Lou, J.-G., Lin, Q., Fu, Q., Ding, R., Zhang, D., and Wang, Z. (2014, January 24–27). Correlating events with time series for incident diagnosis. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA.10.1145/2623330.2623374 ↗
29. Le, T.D., Hoang, T., Li, J., Liu, L., Liu, H., and Hu, S. (2016). A fast PC algorithm for high dimensional causal discovery with multi-core PCs. IEEE/ACM Trans. Comput. Biol. Bioinform., 16.10.1109/TCBB.2016.2591526 ↗