Automatic Parsing and Utilization of System Log Features in Log Analysis: A Survey
Source abstract
Abstract
System logs are almost the only data that records system operation information, so they play an important role in anomaly analysis, intrusion detection, and situational awareness. However, it is still a challenge to obtain effective data from massive system logs. On the one hand, system logs are unstructured data, and, on the other hand, system log records cannot be directly analyzed and calculated by computers. In order to deal with these problems, current researchers digitize system logs through two key steps of log parsing and feature extraction. This paper classifies, analyzes, and summarizes the current log analysis research in terms of log parsing and feature extraction by investigating articles in recent years (including ICSE, TKDD, ICDE, IJCAI, ISSRE, ICDM, ICWS, ICSME, etc.). Finally, in combination with the existing research, the research prospects in the field are elaborated and predicted.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software System Performance and Reliability · Network Security and Intrusion Detection · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210049, China
References (74)
Show source-provided references
1. A survey on automated log analysis for reliability engineering10.1145/3460345 ↗
2. (2022, July 21). Broken Connection in Recent Deployment Causes Microsoft Teams Outage. Available online: https://techwireasia.com/2022/07/broken-connection-in-recent-deployment-causes-microsoft-teams-outage/.
3. (2022, August 19). Google Outage Analysis: 9 August 2022. Available online: https://www.thousandeyes.com/blog/google-outage-analysis-august-9-2022.
4. An SDN-based intrusion detection system using SVM with selective logging for IP traceback10.1016/j.comnet.2021.108015 ↗
5. Cyber intrusion detection through association rule mining on multi-source logs10.1007/s10489-020-02007-5 ↗
6. Multi-datasource machine learning in intrusion detection: Packet flows, system logs and host statistics
7. BCFL logging: An approach to acquire and preserve admissible digital forensics evidence in cloud ecosystem10.1016/j.future.2021.03.001 ↗
8. Digital forensics cloud log unification: Implementing CADF in Apache CloudStack
9. Contextual filtering and prioritization of computer application logs for security situational awareness10.1016/j.future.2019.09.005 ↗
10. Lupton, S., Washizaki, H., Yoshioka, N., and Fukazawa, Y. (2021, January 6–9). Literature Review on Log Anomaly Detection Approaches Utilizing Online Parsing Methodology. Proceedings of the 2021 28th Asia-Pacific Software Engineering Conference (APSEC), Taipei, Taiwan.10.1109/APSEC53868.2021.00068 ↗
11. Zhang, T., Qiu, H., Castellano, G., Rifai, M., Chen, C.S., and Pianese, F. (2023). System Log Parsing: A Survey. IEEE Trans. Knowl. Data Eng., early access.10.1109/TKDE.2022.3222417 ↗
12. Vaarandi, R., and Pihelgas, M. (2015, January 9–13). Logcluster—A data clustering and pattern mining algorithm for event logs. Proceedings of the 2015 11th International Conference on Network and Service Management (CNSM), Barcelona, Spain.10.1109/CNSM.2015.7367331 ↗
13. Hamooni, H., Debnath, B., Xu, J., Zhang, H., Jiang, G., and Mueen, A. (2016, January 24–28). Logmine: Fast pattern recognition for log analytics. Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, Indianapolis, IN, USA.10.1145/2983323.2983358 ↗
14. Du, M., Li, F., Zheng, G., and Srikumar, V. (November, January 30). Deeplog: Anomaly detection and diagnosis from system logs through deep learning. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, USA.10.1145/3133956.3134015 ↗
15. Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., Chen, Y., Zhang, R., Tao, S., and Sun, P. (2019, January 10–16). LogAnomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs. Proceedings of the IJCAI, Macao, China.10.24963/ijcai.2019/658 ↗
16. Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., and Li, Z. (2019, January 26–30). Robust log-based anomaly detection on unstable log data. Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Tallinn, Estonia.10.1145/3338906.3338931 ↗
17. Hitanomaly: Hierarchical transformers for anomaly detection in system log10.1109/TNSM.2020.3034647 ↗
18. Li, X., Chen, P., Jing, L., He, Z., and Yu, G. (2020, January 12–15). Swisslog: Robust and unified deep learning based log anomaly detection for diverse faults. Proceedings of the 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), Coimbra, Portugal.10.1109/ISSRE5003.2020.00018 ↗
19. Xiao, T., Quan, Z., Wang, Z.J., Zhao, K., and Liao, X. (2020, January 17–20). Lpv: A log parser based on vectorization for offline and online log parsing. Proceedings of the 2020 IEEE International Conference on Data Mining (ICDM), Sorrento, Italy.10.1109/ICDM50108.2020.00175 ↗
20. Fang, L., Di, X., Liu, X., Qin, Y., Ren, W., and Ding, Q. (2021, January 20–22). QuickLogS: A Quick Log Parsing Algorithm based on Template Similarity. Proceedings of the 2021 IEEE 20th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), Shenyang, China.10.1109/TrustCom53373.2021.00148 ↗
21. Du, M., and Li, F. (2016, January 12–15). Spell: Streaming parsing of system event logs. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.10.1109/ICDM.2016.0103 ↗
22. He, P., Zhu, J., Zheng, Z., and Lyu, M.R. (2017, January 25–30). Drain: An online log parsing approach with fixed depth tree. Proceedings of the 2017 IEEE International Conference on Web Services (ICWS), Honolulu, HI, USA.10.1109/ICWS.2017.13 ↗
23. Wang, X., Zhao, Y., Xiao, H., Wang, X., and Chi, X. (2021). Ltmatch: A method to abstract pattern from unstructured log. Appl. Sci., 11.10.3390/app11115302 ↗
24. Vervaet, A., Chiky, R., and Callau-Zori, M. (2021, January 7–10). Ustep: Unfixed search tree for efficient log parsing. Proceedings of the 2021 IEEE International Conference on Data Mining (ICDM), Auckland, New Zealand.10.1109/ICDM51629.2021.00077 ↗
25. Fu, Y., Yan, M., Xu, J., Li, J., Liu, Z., Zhang, X., and Yang, D. (2022, January 14–18). Investigating and improving log parsing in practice. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Singapore.10.1145/3540250.3558947 ↗
26. Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., and Luan, Z. (2020, January 20–24). Paddy: An event log parsing approach using dynamic dictionary. Proceedings of the NOMS 2020-2020 IEEE/IFIP Network Operations and Management Symposium, Budapest, Hungary.10.1109/NOMS47738.2020.9110435 ↗
27. Messaoudi, S., Panichella, A., Bianculli, D., Briand, L., and Sasnauskas, R. (June, January 27). A search-based approach for accurate identification of log message formats. Proceedings of the 26th Conference on Program Comprehension, Gothenburg, Sweden.
28. A survey of sequential pattern mining
29. Frequent itemset mining: A 25 years review10.1002/widm.1329 ↗
30. Vaarandi, R. (2003, January 3). A data clustering algorithm for mining patterns from event logs. Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003) (IEEE Cat. No. 03EX764), Kansas City, MO, USA.
31. Nagappan, M., and Vouk, M.A. (2010, January 2–3). Abstracting log lines to log event types for mining software system logs. Proceedings of the 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010), Cape Town, South Africa.10.1109/MSR.2010.5463281 ↗
32. Logram: Efficient Log Parsing Using n n-Gram Dictionaries
33. Sedki, I., Hamou-Lhadj, A., Ait-Mohamed, O., and Shehab, M.A. (2022, January 3–7). An Effective Approach for Parsing Large Log Files. Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME), Limassol, Cyprus.10.1109/ICSME55016.2022.00009 ↗
34. Chu, G., Wang, J., Qi, Q., Sun, H., Tao, S., and Liao, J. (2021, January 19–22). Prefix-Graph: A Versatile Log Parsing Approach Merging Prefix Tree with Probabilistic Graph. Proceedings of the 2021 IEEE 37th International Conference on Data Engineering (ICDE), Chania, Greece.10.1109/ICDE51399.2021.00274 ↗
35. Logstamp: Automatic online log parsing based on sequence labelling10.1145/3543146.3543168 ↗
36. Rand, J., and Miranskyy, A. (2021, January 25–28). On automatic parsing of log records. Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), Madrid, Spain.10.1109/ICSE-NIER52604.2021.00017 ↗
37. Zhang, S., and Wu, G. (2021). Efficient Online Log Parsing with Log Punctuations Signature. Appl. Sci., 11.10.3390/app112411974 ↗
38. Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., and Lyu, M.R. (2019, January 25–31). Tools and benchmarks for automated log parsing. Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), Montreal, QC, Canada.10.1109/ICSE-SEIP.2019.00021 ↗
39. He, S., Zhu, J., He, P., and Lyu, M.R. (2016, January 23–27). Experience report: System log analysis for anomaly detection. Proceedings of the 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), Ottawa, ON, Canada.10.1109/ISSRE.2016.21 ↗
40. Xie, Y., Zhang, H., Zhang, B., Babar, M.A., and Lu, S. (2021, January 22–25). LogDP: Combining Dependency and Proximity for Log-Based Anomaly Detection. Proceedings of the Service-Oriented Computing: 19th International Conference, ICSOC 2021, Virtual Event.10.1007/978-3-030-91431-8_47 ↗
41. Zhao, N., Wang, H., Li, Z., Peng, X., Wang, G., Pan, Z., Wu, Y., Feng, Z., Wen, X., and Zhang, W. (2021, January 23–28). An empirical investigation of practical log anomaly detection for online service systems. Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece.10.1145/3468264.3473933 ↗
42. Lu, S., Wei, X., Li, Y., and Wang, L. (2018, January 12–15). Detecting anomaly in big data system logs using convolutional neural network. Proceedings of the 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech), Athens, Greece.10.1109/DASC/PiCom/DataCom/CyberSciTec.2018.00037 ↗
43. Yen, S., Moh, M., and Moh, T.S. (2019, January 16–19). Causalconvlstm: Semi-supervised log anomaly detection through sequence modeling. Proceedings of the 2019 18th IEEE International Conference On Machine Learning Furthermore, Applications (ICMLA), Boca Raton, FL, USA.10.1109/ICMLA.2019.00217 ↗
44. Bertero, C., Roy, M., Sauvanaud, C., and Trédan, G. (2017, January 23–26). Experience report: Log mining using natural language processing and application to anomaly detection. Proceedings of the 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE), Toulouse, France.10.1109/ISSRE.2017.43 ↗
45. Wang, J., Tang, Y., He, S., Zhao, C., Sharma, P.K., Alfarraj, O., and Tolba, A. (2020). LogEvent2vec: LogEvent-to-vector based anomaly detection for large-scale logs in internet of things. Sensors, 20.10.3390/s20092451 ↗
46. An improved KNN-based efficient log anomaly detection method with automatically labeled samples10.1145/3441448 ↗
47. Lv, D., Luktarhan, N., and Chen, Y. (2021). ConAnomaly: Content-based anomaly detection for system logs. Sensors, 21.10.3390/s21186125 ↗
48. Yang, H., Zhao, X., Sun, D., Wang, Y., and Huang, W. (2021, January 22–25). Sprelog: Log-Based Anomaly Detection with Self-matching Networks and Pre-trained Models. Proceedings of the Service-Oriented Computing: 19th International Conference, ICSOC 2021, Virtual Event.10.1007/978-3-030-91431-8_50 ↗
49. Ryciak, P., Wasielewska, K., and Janicki, A. (2022). Anomaly detection in log files using selected natural language processing methods. Appl. Sci., 12.10.3390/app12105089 ↗
50. Li, Y., Du, N., and Bengio, S. (2017). Time-dependent representation for neural event sequence prediction. arXiv.
51. Rak, T., and Żyła, R. (2022). Using Data Mining Techniques for Detecting Dependencies in the Outcoming Data of a Web-Based System. Appl. Sci., 12.10.3390/app12126115 ↗
52. Xiao, R., Chen, H., Lu, J., Li, W., and Jin, S. (2022). AllInfoLog: Robust Diverse Anomalies Detection Based on All Log Features. IEEE Trans. Netw. Serv. Manag., early access.10.1109/TNSM.2022.3224974 ↗
53. Backes, M., Humbert, M., Pang, J., and Zhang, Y. (November, January 30). walk2friends: Inferring social links from mobility profiles. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, USA.10.1145/3133956.3133972 ↗
54. Dai, H., Dai, B., and Song, L. (2016, January 20–22). Discriminative embeddings of latent variable models for structured data. Proceedings of the International Conference on Machine Learning, PMLR, New York, NY, USA.
55. Xu, X., Liu, C., Feng, Q., Yin, H., Song, L., and Song, D. (November, January 30). Neural network-based graph embedding for cross-platform binary code similarity detection. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, USA.10.1145/3133956.3134018 ↗
56. Perozzi, B., Al-Rfou, R., and Skiena, S. (2014, January 24–27). Deepwalk: Online learning of social representations. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA.10.1145/2623330.2623732 ↗
57. Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., and Mei, Q. (2015, January 18–22). Line: Large-scale information network embedding. Proceedings of the 24th International Conference on World Wide Web, Florence, Italy.10.1145/2736277.2741093 ↗
58. Grover, A., and Leskovec, J. (2016, January 13–17). node2vec: Scalable feature learning for networks. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939754 ↗
59. A markov random field based approach for analyzing supercomputer system logs10.1109/TCC.2017.2678473 ↗
60. Zhao, X., Rodrigues, K., Luo, Y., Yuan, D., and Stumm, M. (2016, January 2–4). Non-Intrusive Performance Profiling for Entire Software Stacks Based on the Flow Reconstruction Principle. Proceedings of the Osdi, Savannah, GA, USA.
61. Milajerdi, S.M., Gjomemo, R., Eshete, B., Sekar, R., and Venkatakrishnan, V. (2019, January 19–23). Holmes: Real-time apt detection through correlation of suspicious information flows. Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA.10.1109/SP.2019.00026 ↗
62. Liu, F., Wen, Y., Zhang, D., Jiang, X., Xing, X., and Meng, D. (2019, January 11–15). Log2vec: A heterogeneous graph embedding based approach for detecting cyber threats within enterprise. Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, London, UK.10.1145/3319535.3363224 ↗
63. Yang, W., Gao, P., Huang, H., Wei, X., Liu, W., Zhu, S., and Luo, W. (2022, January 11–14). RShield: A refined shield for complex multi-step attack detection based on temporal graph network. Proceedings of the Database Systems for Advanced Applications: 27th International Conference, DASFAA 2022, Virtual Event. Proceedings, Part III.10.1007/978-3-031-00129-1_40 ↗
64. Zhang, C., Peng, X., Sha, C., Zhang, K., Fu, Z., Wu, X., Lin, Q., and Zhang, D. (2022, January 21–29). DeepTraLog: Trace-log combined microservice anomaly detection through graph-based deep learning. Proceedings of the 44th International Conference on Software Engineering, Pittsburgh, PA, USA.10.1145/3510003.3510180 ↗
65. Guo, H., Yuan, S., and Wu, X. (2021, January 18–22). Logbert: Log anomaly detection via bert. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.10.1109/IJCNN52387.2021.9534113 ↗
66. Guo, Y., Wu, Y., Zhu, Y., Yang, B., and Han, C. (2021, January 18–22). Anomaly detection using distributed log data: A lightweight federated learning approach. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.10.1109/IJCNN52387.2021.9533294 ↗
67. Chen, R., Zhang, S., Li, D., Zhang, Y., Guo, F., Meng, W., Pei, D., Zhang, Y., Chen, X., and Liu, Y. (2020, January 12–15). Logtransfer: Cross-system log anomaly detection for software systems with transfer learning. Proceedings of the 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), Coimbra, Portugal.10.1109/ISSRE5003.2020.00013 ↗
68. Xia, B., Yin, J., Xu, J., and Li, Y. (2019, January 9–11). Loggan: A sequence-based generative adversarial network for anomaly detection based on system logs. Proceedings of the Science of Cyber Security: Second International Conference, SciSec 2019, Nanjing, China. Revised Selected Papers 2.10.1007/978-3-030-34637-9_5 ↗
69. Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., and Zhang, W. (2021, January 25–28). Semi-supervised log-based anomaly detection via probabilistic label estimation. Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), Madrid, Spain.10.1109/ICSE43902.2021.00130 ↗
70. Le, V.H., and Zhang, H. (2021, January 15–19). Log-based anomaly detection without log parsing. Proceedings of the 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), Melbourne, Australia.10.1109/ASE51524.2021.9678773 ↗
71. Lee, Y., Kim, J., and Kang, P. (2021). LAnoBERT: System log anomaly detection based on BERT masked language model. arXiv.
72. Vervaet, A. (2021, January 19–22). MoniLog: An Automated Log-Based Anomaly Detection System for Cloud Computing Infrastructures. Proceedings of the 2021 IEEE 37th International Conference on Data Engineering (ICDE), Chania, Greece.10.1109/ICDE51399.2021.00317 ↗
73. He, S., Zhu, J., He, P., and Lyu, M.R. (2020). Loghub: A large collection of system log datasets towards automated log analytics. arXiv.
74. Landauer, M., Skopik, F., Höld, G., and Wurzenberger, M. (2022, January 5–7). A User and Entity Behavior Analytics Log dataset for Anomaly Detection in Cloud Computing. Proceedings of the 2022 IEEE International Conference on Big Data (Big Data), Naples, Italy.10.1109/BigData55660.2022.10020672 ↗