Deep Learning-Based Log Parsing for Monitoring Industrial ICT Systems
Source abstract
Abstract
For rapidly developing smart manufacturing, Industrial ICT Systems (IICTSs) have become critical to safe and reliable production, and effective monitoring of complex IICTSs in practice is necessary but challenging. Since such monitoring data are organized generally as semi-structural logs, log parsing, the fundamental premise of advanced log analysis, has to be comprehensively addressed. Because of unrealistic assumptions, high maintenance costs, and the incapability of distinguishing homologous logs, existing log parsing methods cannot simultaneously fulfill the requirements of complex IICTSs simultaneously. Focusing on these issues, we present LogParser, a deep learning-based framework for both online and offline parsing of IICTS logs. For performance evaluation, we conduct extensive experiments based on monitoring log sets from 18 different real-world systems. The results demonstrate that LogParser achieves at least a 14.5% higher parsing accuracy than the state-of-the-art methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software System Performance and Reliability · Advanced Software Engineering Methodologies · Network Security and Intrusion Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
National Engineering Laboratory for Big Data Analytics, Xi’an Jiaotong University, Xi’an 710049, China
Funding
National Key Research and Development Program of China · National Natural Science Foundation of China · China Postdoctoral Science Foundation
References (32)
Show source-provided references
1. Lee, J., Bagheri, B., and Kao, H.A. (2014, January 27–30). Recent advances and trends of cyber-physical systems and big data analytics in industrial informatics. Proceedings of the 12th International Conference on Industrial Informatics (INDIN), Porto Alegre, Brazil.
2. A CNN-based visual sorting system with cloud-edge computing for flexible manufacturing systems10.1109/TII.2019.2947539 ↗
3. A manufacturing big data solution for active preventive maintenance10.1109/TII.2017.2670505 ↗
4. He, P., Zhu, J., He, S., Li, J., and Lyu, M.R. (July, January 28). An evaluation study on log parsing and its use in log mining. Proceedings of the 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), Toulouse, France.
5. Xu, W., Huang, L., Fox, A., Patterson, D., and Jordan, M.I.T. (2009, January 11–14). Detecting large-scale system problems by mining console logs. Proceedings of the 22nd ACM Symposium on Operating Systems Principles, Big Sky, MT, USA.10.1145/1629575.1629587 ↗
6. (2020, August 04). Samples of Security Related Data. Available online: https://www.secrepo.com/.
7. (2020, August 04). Thingworx Platform. Available online: https://developer.thingworx.com/en/platform.
8. Vaarandi, R. (2003, January 3). A data clustering algorithm for mining patterns from event logs. Proceedings of the IEEE IPOM Workshop, Kansas City, MO, USA.
9. Tang, L., Li, T., and Perng, C.S. (2011, January 24–28). LogSig: Generating system events from raw textual logs. Proceedings of the 20th ACM International Conference on Information and Knowledge Management, Glasgow, UK.10.1145/2063576.2063690 ↗
10. Fu, Q., Lou, J.G., Wang, Y., and Li, J. (2009, January 6–9). Execution anomaly detection in distributed systems through unstructured log analysis. Proceedings of the IEEE ICDM, Miami Beach, FL, USA.10.1109/ICDM.2009.60 ↗
11. Makanju, A.A., Zincir-Heywood, A.N., and Milios, E.E. (July, January 28). Clustering event logs using iterative partitioning. Proceedings of the The 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France.
12. He, P., Zhu, J., Zheng, Z., and Lyu, M.R. (2017, January 25–30). Drain: An online log parsing approach with fixed depth tree. Proceedings of the 2017 IEEE International Conference on Web Services (ICWS), Honolulu, HI, USA.10.1109/ICWS.2017.13 ↗
13. Towards automated log parsing for large-scale log data analysis10.1109/TDSC.2017.2762673 ↗
14. Aussel, N., Petetin, Y., and Chabridon, S. (2018, January 25–28). Improving performances of log mining for anomaly prediction through nlp-based log parsing. Proceedings of the International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), Milwaukee, WI, USA.10.1109/MASCOTS.2018.00031 ↗
15. Nguyen, T., Kobayashi, S., and Fukuda, K. (2021, January 17–21). Logdtl: Network log template generation with deep transfer learning. Proceedings of the 17th IFIP/IEEE International Symposium on Integrated Network Management, Bordeaux, France.
16. Nedelkoski, S., Bogatinovski, J., Acker, A., Cardoso, J., and Kao, O. (2021, January 13–17). Self-supervised log parsing. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Bilbao, Spain.10.1007/978-3-030-67667-4_8 ↗
17. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013, January 2–4). Efficient estimation of word representations in vector space. Proceedings of the Conference on Learning Representations, ICLR 2013, Scottsdale, AZ, USA.
18. Dropout: A simple way to prevent neural networks from overfitting
19. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
20. Framewise phoneme classification with bidirectional LSTM and other neural network architectures10.1016/j.neunet.2005.06.042 ↗
21. Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7–9). Neural machine translation by jointly learning to align and translate. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA.
22. Data driven bottleneck detection of manufacturing systems10.1080/00207540701881860 ↗
23. Ji, C., Liu, S., Yang, C., Wu, L., and Pan, L. (2015, January 22–23). Ibdp: An industrial big data ingestion and analysis platform and case studies. Proceedings of the International Conference on Identification, Information and Knowledge in the Internet of Things (IIKI), Beijing, China.10.1109/IIKI.2015.55 ↗
24. Multilayer data-driven cyber-attack detection system for industrial control systems based on network, system, and process data10.1109/TII.2019.2891261 ↗
25. A hybrid computing solution and resource scheduling strategy for edge computing in smart manufacturing10.1109/TII.2019.2899679 ↗
26. Oliner, A., and Stearley, J. (2007, January 25–28). What supercomputers say: A study of five system logs. Proceedings of the International Conference on Dependable Systems and Networks (DSN), Edinburgh, UK.10.1109/DSN.2007.103 ↗
27. Web server workload characterization: The search for invariants10.1145/233008.233034 ↗
28. Du, M., Li, F., Zheng, G., and Srikumar, V. (November, January 30). Deeplog: Anomaly detection and diagnosis from system logs through deep learning. Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, USA.
29. Sconzo, M., and Dorsey, D. (2020, August 04). Connection Log. Security Data Analysis Labs. Available online: https://github.com/sooshie/Security-Data-Analysis.
30. Committee, I.V.C. (2020, August 04). Big Brother Data. Available online: http://vacommunity.org/VAST+Challenge+2013.
31. Industrial cyber security conference 4SICS, T. (2020, August 04). Capture Files from 4SICS Geek Lounge. Available online: https://www.netresec.com/index.ashx?page=PCAP4SICS.
32. Miettinen, M., Marchal, S., Hafeez, I., Asokan, N., Sadeghi, A.R., and Tarkoma, S. (2017, January 5–8). Iot sentinel: Automated device-type identification for security enforcement in iot. Proceedings of the 37th IEEE International Conference on Distributed Computing Systems, ICDCS 2017, Atlanta, GA, USA.10.1109/ICDCS.2017.283 ↗