DLogParser: An Efficient Dynamic Log Parser with Multiple Grouping Criteria
Source abstract
Abstract
Most existing log parsers are static. When parsing logs with a static parser, the accuracy tends to fluctuate significantly. To overcome this issue, this paper proposes a dynamic log parser named DLogParser. The core idea of DLogParser is to select different parsing policies based on log features. DLogParser first parses a small batch of sample log messages, then analyzes log characteristics from the parsing results, and determines an appropriate parsing policy for the current logs. Then it parses all remaining logs according to the determined policy. To support dynamic parsing policies, DLogParser incorporates 5 grouping criteria for log features, including length, punctuation, first token, last token, and key token, and establishes 7 rules for parsing policy generation. We evaluated DLogParser on public datasets from LogHub. The experimental results demonstrate that compared to 11 existing log parsers, DLogParser achieves an accuracy of 90.3% with an acceptable performance loss.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software System Performance and Reliability · Cloud Computing and Resource Management · IoT and Edge/Fog Computing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, Zhongyuan University of Technology, Zhengzhou 450007, China
Zhengzhou Information Science and Technology Institute, Zhengzhou 450001, China
Funding
National Natural Science Foundation of China · Henan Provincial Key Scientific and Technological Research Project
References (35)
Show source-provided references
1. A Survey on Automated Log Analysis for Reliability Engineering10.1145/3460345 ↗
2. Le, V.-H., and Zhang, H.Y. (2022, January 25–27). Log-based Anomaly Detection with Deep Learning: How Far Are We?. Proceedings of the IEEE/ACM International Conference on Software Engineering, Pittsburgh, PA, USA.10.1145/3510003.3510155 ↗
3. Yang, L., Chen, J.J., Wang, Z., Wang, W.J., Jiang, J.J., and Dong, X.Y. (2021, January 22–30). Semi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation. Proceedings of the IEEE/ACM International Conference on Software Engineering, Madrid, Spain.10.1109/ICSE43902.2021.00130 ↗
4. Pathidea: Improving Information Retrieval-Based Bug Localization by Re-Constructing Execution Paths Using Logs10.1109/TSE.2021.3071473 ↗
5. Chuah, E., Kuo, S.-H., Hiew, P., Tjhi, W.-C., Lee, G., and Hammond, J. (2010, January 19–22). Diagnosing the root-causes of failures from cluster log files. Proceedings of the International Conference on High Performance Computing, Goa, India.10.1109/HIPC.2010.5713159 ↗
6. LogRule: Efficient Structured Log Mining for Root Cause Analysis10.1109/TNSM.2023.3282270 ↗
7. Bushong, V., Sanders, R., Curtis, J., Du, M., Cerny, T., Frajtak, K., Bures, M., Tisnovsky, P., and Shin, D.W. (2020, January 13–16). On Matching Log Analysis to Source Code: A Systematic Mapping Study. Proceedings of the International Conference on Research in Adaptive and Convergent Systems, Gwangju, Republic of Korea.10.1145/3400286.3418262 ↗
8. Shang, W.Y. (2012, January 2–9). Bridging the divide between software developers and operators using logs. Proceedings of the International Conference on Software Engineering, Zurich, Switzerland.10.1109/ICSE.2012.6227031 ↗
9. Vaarandi, R. (2003, January 1–3). A data clustering algorithm for mining patterns from event logs. Proceedings of the IEEE Workshop on IP Operations and Management, Kansas City, MO, USA.
10. Mining Frequent Patterns without Candidate Generation: A Frequent-Pattern Tree Approach10.1023/B:DAMI.0000005258.31418.83 ↗
11. Vaarandi, R., and Pihelgas, M. (2015, January 9–13). LogCluster—A data clustering and pattern mining algorithm for event logs. Proceedings of the International Conference on Network and Service Management, Barcelona, Spain.10.1109/CNSM.2015.7367331 ↗
12. Hamooni, H., Debnath, B., Xu, J.W., Zhang, H., Jiang, G.F., and Mueen, A. (2016, January 24–28). LogMine: Fast Pattern Recognition for Log Analytics. Proceedings of the ACM International on Conference on Information and Knowledge Management, Indianapolis, IN, USA.10.1145/2983323.2983358 ↗
13. Fu, Q., Lou, J.-G., Wang, Y., and Li, J. (2009, January 6–9). Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis. Proceedings of the IEEE International Conference on Data Mining, Miami, FL, USA.10.1109/ICDM.2009.60 ↗
14. Tang, L., Li, T., and Perng, C.-S. (2011, January 24–28). LogSig: Generating system events from raw textual logs. Proceedings of the ACM international conference on Information and knowledge management, Glasgow, UK.10.1145/2063576.2063690 ↗
15. Jiang, Z.M., Hassan, A.E., Flora, P., and Hamann, G. (2008, January 12–13). Abstracting Execution Logs to Execution Events for Enterprise Applications. Proceedings of the International Conference on Quality Software, Oxford, UK.10.1109/QSIC.2008.50 ↗
16. Makanju, A.A.O., Zincir-Heywood, A.N., and Milios, E.E. (July, January 28). Clustering event logs using iterative partitioning. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France.
17. He, P., Zhu, J., Zheng, Z., and Lyu, M.R. (2017, January 25–30). Drain: An Online Log Parsing Approach with Fixed Depth Tree. Proceedings of the IEEE International Conference on Web Services, Honolulu, HI, USA.10.1109/ICWS.2017.13 ↗
18. Le, V.-H., and Zhang, H. (2023, January 14–20). Log Parsing with Prompt-based Few-shot Learning. Proceedings of the IEEE/ACM International Conference on Software Engineering, Australia.10.1109/ICSE48619.2023.00204 ↗
19. LILAC: Log Parsing using LLMs with Adaptive Parsing Cache10.1145/3643733 ↗
20. Xu, A., and Gau, A. (2024). HELP: Hierarchical Embeddings-based Log Parsing. arXiv.
21. He, P.J., Zhu, J.M., Xu, P.C., Zheng, Z.B., and Lyu, M.R. (2018). A Directed Acyclic Graph Approach to Online Log Parsing. arXiv.
22. Yuan, J.H., Zhou, H.W., Wang, C., and Guan, B. (2024, January 13–16). nDrain: A Robust Log Template Mining Algorithm. Proceedings of the International Conference on Computer and Communications, Chengdu, China.10.1109/ICCC62609.2024.10941951 ↗
23. Zhu, J.M., He, S.L., Liu, J.Y., He, P.J., Xie, Q., and Zheng, Z.B. (2019, January 25–31). Tools and Benchmarks for Automated Log Parsing. Proceedings of the IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, Montreal, QC, Canada.10.1109/ICSE-SEIP.2019.00021 ↗
24. He, P.J., Zhu, J.M., He, S.L., Li, J., and Lyu, M.R. (July, January 28). An Evaluation Study on Log Parsing and Its Use in Log Mining. Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks, Toulouse, France.
25. PosParser: A Heuristic Online Log Parsing Method Based on Part-of-Speech Tagging10.1109/TBDATA.2024.3453756 ↗
26. Zhu, J.M., He, S.L., He, P.J., Liu, J.Y., and Lyu, M.R. (2023, January 9–12). Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics. Proceedings of the IEEE International Symposium on Software Reliability Engineering, Florence, Italy.10.1109/ISSRE59848.2023.00071 ↗
27. Nagappan, M., and Vouk, M.A. (2010, January 2–3). Abstracting log lines to log event types for mining software system logs. Proceedings of the IEEE Working Conference on Mining Software Repositories, Cape Town, South Africa.10.1109/MSR.2010.5463281 ↗
28. Du, M., and Li, F.F. (2016, January 12–15). Spell: Streaming Parsing of System Event Logs. Proceedings of the IEEE International Conference on Data Mining, Barcelona, Spain.10.1109/ICDM.2016.0103 ↗
29. Wu, Y., Yu, S., and Li, Y. (2025). Log Parsing using LLMs with Self-Generated In-Context Learning and Self-Correction. arXiv.10.1109/ICPC66645.2025.00063 ↗
30. Zhang, S.J., and Gang, W. (2021). Efficient Online Log Parsing with Log Punctuations Signature. Appl. Sci., 11.10.3390/app112411974 ↗
31. Jiang, Z.H., Liu, J.Y., Huang, J.J., Li, Y.C., Huo, Y.T., Gu, J.Z., Chen, Z.B., Zhu, J.M., and Lyu, M.R. (2024, January 16–20). A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?. Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis, New York, NY, USA.10.1145/3650212.3652123 ↗
32. Ma, Z., Chen, A.R., Kim, D.J., Chen, T.-H.P., and Wang, S. (2024, January 14–20). LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing. Proceedings of the IEEE/ACM International Conference on Software Engineering, Lisbon, Portugal.10.1145/3597503.3639150 ↗
33. Xu, J.J.L., Yang, R.C., Huo, Y.T., Zhang, C.Y., and He, P.J. (2024, January 14–20). DivLog: Log Parsing with Prompt Enhanced In-Context Learning. Proceedings of the IEEE/ACM International Conference on Software Engineering, Lisbon, Portugal.10.1145/3597503.3639155 ↗
34. Ma, Z.Y., Kim, D.J., and Chen, T.-H.P. (May, January 26). LibreLog: Accurate and Efficient Unsupervised Log Parsing Using Open-Source Large Language Models. Proceedings of the IEEE/ACM International Conference on Software Engineering, Ottawa, ON, Canada.
35. Xiao, Y., Le, V.-H., and Zhang, H.Y. (November, January 27). Demonstration-Free: Towards More Practical Log Parsing with Large Language Models. Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, Sacramento, CA, USA.