The Diagnosis-Effective Sampling of Application Traces
Source abstract
Abstract
Distributed tracing is cutting-edge technology used for monitoring, managing, and troubleshooting native cloud applications. It offers a more comprehensive and continuous observability, surpassing traditional logging methods, and is indispensable for navigating modern complex software architectures. However, the sheer volume of generated traces is staggering in distributed applications, and the direct storage and utilization of every trace is impractical due to associated operational costs. This entails a sampling strategy to select which traces warrant storage and analysis. Historically, sampling methods have included a rate-based approach, often relying heavily on a manual configuration. There is a need for a more intelligent approach, and we propose a hierarchical sampling methodology to address multiple requirements concurrently. Initial rate-based sampling mitigates the overwhelming volume of traces, as no further analysis can be performed on this level. In the next stage, more nuanced analysis is facilitated based on the previous foundation, incorporating information regarding trace properties and ensuring the preservation of vital process details even under extreme conditions. This comprehensive approach not only aids in the visualization and conceptualization of applications but also enables more targeted analysis in later stages. As we delve deeper into the sampling hierarchy, the technique becomes tailored to specific purposes, such as the simplification of application troubleshooting. In this context, the sampling strategy prioritizes the retention of erroneous traces from dominant processes, thus facilitating the identification and resolution of underlying issues. The focus of this paper is to reveal the impact of sampling on troubleshooting efficiency. Leveraging intelligent and explainable artificial intelligence solutions enables the detection of malfunctioning microservices and provides transparent insights into root causes. We advocate for using rule-induction systems, which offer explainability and efficacy in decision-making processes. By integrating advanced sampling techniques with machine-learning-driven intelligence, we empower organizations to navigate the complexities of large-scale distributed cloud environments effectively.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software System Performance and Reliability · Software Engineering Research · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Mathematics NAS Armenia, Yerevan 0019, Armenia
College of Science and Engineering, American University of Armenia, Yerevan 0019, Armenia
ML Laboratory, Yerevan State University, Yerevan 0025, Armenia
Picsart, Miami, FL 33009, USA
Department of Computer Science, University of Chile, Santiago 8330111, Chile
Funding
ADVANCE Research Grants from the Foundation for Armenian Science and Technology
References (76)
Show source-provided references
1. Parker, A., Spoonhower, D., Mace, J., Sigelman, B., and Isaacs, R. (2020). Distributed Tracing in Practice: Instrumenting, Analyzing, and Debugging Microservices, O’Reilly Media, Incorporated.
2. Shkuro, Y. (2019). Mastering Distributed Tracing: Analyzing Performance in Microservices and Complex Systems, Packt Publishing.
3. Opentracing (2021, January 26). What Is Distributed Tracing?. Available online: https://opentracing.io/docs/overview/what-is-tracing/.
4. A real-time trace-level toot-cause diagnosis system in Alibaba datacenters10.1109/ACCESS.2019.2944456 ↗
5. Liu, D., He, C., Peng, X., Lin, F., Zhang, C., Gong, S., Li, Z., Ou, J., and Wu, Z. (2021, January 25–28). MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems. Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), Madrid, Spain.10.1109/ICSE-SEIP52600.2021.00043 ↗
6. Poghosyan, A.V., Harutyunyan, A.N., Grigoryan, N.M., and Pang, C. (2022, January 23–26). Root Cause Analysis of Application Performance Degradations via Distributed Tracing. Proceedings of the Third CODASSCA Workshop, Yerevan, Armenia: Collaborative Technologies and Data Science in Artificial Intelligence Applications, Yerevan, Armenia.
7. Distributed Tracing for Troubleshooting of Native Cloud Applications via Rule-Induction Systems10.3897/jucs.112513 ↗
8. (2024, June 25). Distributed Tracing—Past, Present and Future. Available online: https://www.zerok.ai/post/distributed-tracing-past-present-future.
9. Young, T., and Parker, A. (2024). Learning OpenTelemetry, O’Reilly Media.
10. Run-time failure detection via non-intrusive event analysis in a large-scale cloud computing platform10.1016/j.jss.2023.111611 ↗
11. Zhang, X., Lin, Q., Xu, Y., Qin, S., Zhang, H., Qiao, B., Dang, Y., Yang, X., Cheng, Q., and Chintalapati, M. (2019, January 10–12). Cross-dataset Time Series Anomaly Detection for Cloud Systems. Proceedings of the 2019 USENIX Annual Technical Conference (USENIX ATC 19), Renton, WA, USA.
12. Abad, C., Taylor, J., Sengul, C., Yurcik, W., Zhou, Y., and Rowe, K. (2003, January 8–12). Log correlation for intrusion detection: A proof of concept. Proceedings of the 19th Annual Computer Security Applications Conference, Las Vegas, NV, USA.
13. Suriadi, S., Ouyang, C., van der Aalst, W., and ter Hofstede, A. (2012, January 3–6). Root cause analysis with enriched process logs. Proceedings of the Business Process Management Workshops, International Workshop on Business Process Intelligence (BPI 2012), Tallin, Estonia.10.1007/978-3-642-36285-9_18 ↗
14. BigPanda (2021, January 26). Incident Management. Available online: https://docs.bigpanda.io/docs/incident-management.
15. Josefsson, T. (2017). Root-Cause Analysis through Machine Learning in the Cloud. [Master’s Thesis, Uppsala Universitet].
16. Tak, B., Tao, S., Yang, L., Zhu, C., and Ruan, Y. (2016, January 4–8). LOGAN: Problem diagnosis in the cloud using log-based reference models. Proceedings of the 2016 IEEE International Conference on Cloud Engineering (IC2E), Berlin, Germany.10.1109/IC2E.2016.12 ↗
17. Localizing root causes of performance anomalies in cloud computing systems by analyzing request trace logs10.1007/s11432-012-4747-8 ↗
18. Poghosyan, A., Harutyunyan, A., Grigoryan, N., Pang, C., Oganesyan, G., Ghazaryan, S., and Hovhannisyan, N. (2021). An Enterprise Time Series Forecasting System for Cloud Applications Using Transfer Learning. Sensors, 21.10.20944/preprints202101.0326.v1 ↗
19. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI10.1016/j.inffus.2019.12.012 ↗
20. Cohen, W.W. (1995, January 9–12). Fast Effective Rule Induction. Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, CA, USA.
21. Quinlan, J.R. (2014). C4.5: Programs for Machine Learning, Elsevier.
22. (2024, June 25). Jaeger: Sampling. Available online: https://www.jaegertracing.io/docs/1.55/sampling/.
23. (2024, June 25). Anomaly Detection in Zipkin Trace Data. Available online: https://engineering.salesforce.com/anomaly-detection-in-zipkin-trace-data-87c8a2ded8a1/.
24. (2024, June 25). LightStep: Sampling, Verbosity, and the Case for (Much) Broader Applications of Distributed Tracing. Available online: https://medium.com/lightstephq/sampling-verbosity-and-the-case-for-much-broader-applications-of-distributed-tracing-f3500a174c17.
25. (2024, June 25). Datadog: Trace Sampling Use Cases. Available online: https://docs.datadoghq.com/tracing/guide/ingestion_sampling_use_cases/.
26. (2024, June 25). Partial Trace Sampling: A New Approach to Distributed Trace Sampling. Available online: https://engineering.dynatrace.com/blog/partial-trace-sampling-a-new-approach-to-distributed-trace-sampling/.
27. (2024, June 25). New Relic: Technical Distributed Tracing Details. Available online: https://docs.newrelic.com/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works/#sampling.
28. (2024, June 25). OpenTelemetry Trace Sampling. Available online: https://docs.appdynamics.com/observability/cisco-cloud-observability/en/application-performance-monitoring/opentelemetry-trace-sampling.
29. (2024, June 25). When to Sample. Available online: https://docs.honeycomb.io/manage-data-volume/sample/guidelines/.
30. (2024, June 25). An Introduction to Trace Sampling with Grafana Tempo and Grafana Agent. Available online: https://grafana.com/blog/2022/05/11/an-introduction-to-trace-sampling-with-grafana-tempo-and-grafana-agent/.
31. (2024, June 25). Application Performance Monitoring: Transaction Sampling. Available online: https://www.elastic.co/guide/en/observability/current/apm-sampling.html.
32. Las-Casas, P., Papakerashvili, G., Anand, V., and Mace, J. (2019, January 20–23). Sifter: Scalable Sampling for Distributed Traces, without Feature Engineering. Proceedings of the ACM Symposium on Cloud Computing, New York, NY, USA.10.1145/3357223.3362736 ↗
33. Thereska, E., Salmon, B., Strunk, J., Wachs, M., Abd-El-Malek, M., Lopez, J., and Ganger, G.R. (2006, January 26–30). Stardust: Tracking activity in a distributed storage system. Proceedings of the Joint International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS’06), Saint-Malo, France.10.1145/1140277.1140280 ↗
34. Sambasivan, R.R., Zheng, A.X., Rosa, M.D., Krevat, E., Whitman, S., Stroucken, M., Wang, W., Xu, L., and Ganger, G.R. (March, January 1). Diagnosing performance changes by comparing request flows. Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI), Boston, MA, USA.
35. Fonseca, R., Porter, G., Katz, R.H., Shenker, S., and Stoica, I. (2007, January 11–13). X-trace: A pervasive network tracing framework. Proceedings of the 4th USENIX Conference on Networked Systems Design & Implementation, Cambridge, MA, USA.
36. Sigelman, B.H., Barroso, L.A., Burrows, M., Stephenson, P., Plakal, M., Beaver, D., Jaspan, S., and Shanbhag, C. (2010). Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Google, Inc.. Technical Report.
37. Kaldor, J., Mace, J., Bejda, M., Gao, E., Kuropatwa, W., O’Neill, J., Ong, K.W., Schaller, B., Shan, P., and Viscomi, B. (2017, January 28–31). Canopy: An End-to-End Performance Tracing And Analysis System. Proceedings of the 26th Symposium on Operating Systems Principles, Shanghai, China.
38. (2024, June 25). OpenTelemetry. Available online: https://opentelemetry.io/.
39. Las-Casas, P., Mace, J., Guedes, D., and Fonseca, R. (2018, January 11–13). Weighted Sampling of Execution Traces: Capturing More Needles and Less Hay. Proceedings of the ACM Symposium on Cloud Computing, Carlsbad, CA, USA.10.1145/3267809.3267841 ↗
40. (2024, June 25). Google Cloud Observability: Trace Sampling. Available online: https://cloud.google.com/trace/docs/trace-sampling.
41. (2024, June 25). OpenCensus: Sampling. Available online: https://opencensus.io/tracing/sampling/.
42. (2024, June 25). Azure Monitor: Sampling in Application Insights. Available online: https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling-classic-api.
43. He, S., Feng, B., Li, L., Zhang, X., Kang, Y., Lin, Q., Rajmohan, S., and Zhang, D. (2023, January 3–9). STEAM: Observability-Preserving Trace Sampling. Proceedings of the FSE’23 Industry, San Francisco, CA, USA.10.1145/3611643.3613881 ↗
44. (2024, June 25). AWS: Advanced Sampling Using ADOT. Available online: https://aws-otel.github.io/docs/getting-started/advanced-sampling#best-practices-for-advanced-sampling.
45. Solé, M., Muntés-Mulero, V., Rana, A.I., and Estrada, G. (2017). Survey on models and techniques for root-cause analysis. arXiv.
46. On machine learning approaches for automated log management
47. Incident Management for Explainable and Automated Root Cause Analysis in Cloud Data Centers10.3897/jucs.76608 ↗
48. Kounev, S., Giese, H., and Liu, J. (2016, January 17–22). Managing cloud infrastructures by a multi-layer data analytics. Proceedings of the 2016 IEEE International Conference on Autonomic Computing, ICAC 2016, Wuerzburg, Germany.
49. Turck, F.D., Diao, Y., Hong, C.S., Medhi, D., and Sadre, R. (2013, January 27–31). Pattern detection in unstructured data: An experience for a virtualized IT infrastructure. Proceedings of the 2013 IFIP/IEEE International Symposium on Integrated Network Management, IM 2013, Ghent, Belgium.
50. Reynolds, P., Killian, C.E., Wiener, J.L., Mogul, J.C., Shah, M.A., and Vahdat, A. (2007, January 8–10). Pip: Detecting the Unexpected in Distributed Systems. Proceedings of the Symposium on Networked Systems Design and Implementation, San Jose, CA, USA.
51. Challenges and Experiences in Designing Interpretable KPI-diagnostics for Cloud Applications10.3897/jucs.112570 ↗
52. Fürnkranz, J., Gamberger, D., and Lavrač, N. (2012). Foundations of Rule Learning, Springer. Cognitive Technologies.10.1007/978-3-540-75197-7 ↗
53. Bassiliades, N., Gottlob, G., Sadri, F., Paschke, A., and Roman, D. (2015, January 2–5). A brief overview of rule learning. Proceedings of the Rule Technologies: Foundations, Tools, and Applications, Berlin, Germany.10.1007/978-3-319-21542-6 ↗
54. Pruning Algorithms for Rule Learning10.1023/A:1007329424533 ↗
55. Fürnkranz, J., and Widmer, G. (1994, January 10–13). Incremental reduced error pruning. Proceedings of the 11th International Conference on Machine Learning, New Brunswick, NJ, USA.
56. FURIA: An algorithm for unordered fuzzy rule induction10.1007/s10618-009-0131-8 ↗
57. Fast Dimensional Analysis for Root Cause Investigation in a Large-Scale Service Environment10.1145/3392149 ↗
58. Lee, W., and Stolfo, S.J. (1998, January 26–29). Data Mining Approaches for Intrusion Detection. Proceedings of the 7th Conference on USENIX Security Symposium, San Antonio, TX, USA.
59. Helmer, G., Wong, J., Honavar, V., and Miller, L. (1998, January 3). Intelligent agents for intrusion detection. Proceedings of the IEEE Information Technology Conference, Syracuse, NY, USA.
60. Automated discovery of concise predictive rules for intrusion detection10.1016/S0164-1212(01)00088-7 ↗
61. Mannila, H., Toivonen, H., and Verkamo, A.I. (1995, January 20–21). Discovering Frequent Episodes in Sequences Extended Abstract. Proceedings of the First International Conference on Knowledge Discovery and Data Mining, Montreal, QC, Canada.
62. Liu, H., and Motoda, H. (1998). Perspectives of Feature Selection. Feature Selection for Knowledge Discovery and Data Mining, Springer.10.1007/978-1-4615-5689-3 ↗
63. John, G.H., Kohavi, R., and Pfleger, K. (1994, January 10–13). Irrelevant features and the subset selection problem. Proceedings of the Machine Learning: Proceedings of the 11th International Conference, New Brunswick, NJ, USA.
64. Agrawal, R., Imieliński, T., and Swami, A. (1993, January 26–28). Mining association rules between sets of items in large databases. Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, Washington, DC, USA.10.1145/170035.170072 ↗
65. Shafer, G. (1976). A Mathematical Theory of Evidence, Princeton University Press.10.1515/9780691214696 ↗
66. Applying Dempster–Shafer theory for developing a flexible, accurate and interpretable classifier10.1016/j.eswa.2020.113262 ↗
67. Poghosyan, A., Harutyunyan, A., Davtyan, E., Petrosyan, K., and Baloian, N. (2024). A Study on Automated Problem Troubleshooting in Cloud Environments with Rule Induction and Verification. Appl. Sci., 14.10.3390/app14031047 ↗
68. Chen, Z., Jiang, Z., Su, Y., Lyu, M.R., and Zheng, Z. (2024). TraceMesh: Scalable and Streaming Sampling for Distributed Traces. arXiv.10.1109/CLOUD62652.2024.00016 ↗
69. Gias, A.U., Gao, Y., Sheldon, M., Perusquía, J.A., O’Brien, O., and Casale, G. (2022). SampleHST: Efficient On-the-Fly Selection of Distributed Traces. arXiv.
70. Huang, Z., Chen, P., Yu, G., Chen, H., and Zheng, Z. (2021, January 5–11). Sieve: Attention-based Sampling of End-to-End Trace Data in Distributed Microservice Systems. Proceedings of the 2021 IEEE International Conference on Web Services (ICWS), Virtual.10.1109/ICWS53863.2021.00063 ↗
71. Zhou, T., Zhang, C., Peng, X., Yan, Z., Li, P., Liang, J., Zheng, H., Zheng, W., and Deng, Y. (2023, January 9–12). TraceStream: Anomalous Service Localization based on Trace Stream Clustering with Online Feedback. Proceedings of the 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), Florence, Italy.10.1109/ISSRE59848.2023.00033 ↗
72. Lundberg, S., and Lee, S.I. (2017). A Unified Approach to Interpreting Model Predictions. arXiv.
73. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. arXiv.10.1145/2939672.2939778 ↗
74. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median10.1016/j.jesp.2013.03.013 ↗
75. The t-digest: Efficient estimates of distributions10.1016/j.simpa.2020.100049 ↗
76. Witten, I.H., Frank, E., Hall, M.A., and Pal, C.J. (2005). Practical Machine Learning Tools and Techniques, Morgan Kaufmann.