Some Improvements of Behavioral Malware Detection Method Using Graph Neural Networks
Source abstract
Abstract
This study proposes improvements to a behavioral malware detection method based on graph convolutional networks (GCNs). Three main modifications were investigated: improved normalization of the adjacency matrix, a multi-layer GCN architecture, and a parallel dual-normalization model. The models were trained on a dataset of 44,000 Windows API call sequences and evaluated using standard metrics—accuracy, precision, recall, F1 score, and ROC AUC. The best performance was achieved by the four-layer GCN, which outperformed the baseline in most metrics. The results also showed a non-monotonic relationship between model quality and network depth, likely caused by over-smoothing effects. This study confirms that properly tuned GCN architectures can significantly improve the accuracy and robustness of malware detection.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Malware Detection Techniques · Network Security and Intrusion Detection · Digital and Cyber Forensics
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Cybernetics, Military University of Technology, Gen. Sylwestra Kaliskiego 2 Street, 00-908 Warsaw, Poland
Funding
Military University of Technology in Warsaw
References (37)
Show source-provided references
1. A Comprehensive Survey on Graph Neural Networks10.1109/TNNLS.2020.2978386 ↗
2. Liu, Z., Chen, C., Yang, X., Zhou, J., Li, X., and Song, L. (2018, January 22–26). Heterogeneous Graph Neural Networks for Malicious Account Detection. Proceedings of the 27th ACM International Conference on Information and Knowledge Management, Torino, Italy.10.1145/3269206.3272010 ↗
3. Wang, J., Wen, R., Wu, C., Huang, Y., and Xiong, J. (2019, January 13–17). FdGars: Fraudster Detection via Graph Convolutional Networks in Online App Review System. Proceedings of the Companion Proceedings of the 2019 World Wide Web Conference, San Francisco, CA, USA.10.1145/3308560.3316586 ↗
4. Schranko De Oliveira, A., and Sassi, R.J. (2019). Behavioral Malware Detection Using Deep Graph Convolutional Neural Networks. TechRxiv.10.36227/techrxiv.10043099.v1 ↗
5. A Novel Approach for Detecting Deep Fake Videos Using Graph Neural Network10.1186/s40537-024-00884-y ↗
6. Dou, Y., Liu, Z., Sun, L., Deng, Y., Peng, H., and Yu, P.S. (2020, January 19–23). Enhancing Graph Neural Network-Based Fraud Detectors against Camouflaged Fraudsters. Proceedings of the 29th ACM International Conference on Information & Knowledge Management, Virtual.10.1145/3340531.3411903 ↗
7. Liu, Z., Dou, Y., Yu, P.S., Deng, Y., and Peng, H. (2020, January 25–30). Alleviating the Inconsistency Problem of Applying Graph Neural Network to Fraud Detection. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual.10.1145/3397271.3401253 ↗
8. BGNN4VD: Constructing Bidirectional Graph Neural-Network for Vulnerability Detection10.1016/j.infsof.2021.106576 ↗
9. DeepWukong: Statically Detecting Software Vulnerabilities Using Deep Graph Neural Network10.1145/3436877 ↗
10. Zhou, Y., Liu, S., Siow, J., Du, X., and Liu, Y. (2019). Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks. arXiv.
11. Song, W., Yin, H., Liu, C., and Song, D. (2018, January 15–19). DeepMem: Learning Graph Neural Network Models for Fast and Robust Memory Forensic Analysis. Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, Toronto, ON, Canada.10.1145/3243734.3243813 ↗
12. Jafari, O., Maurya, P., Nagarkar, P., Islam, K.M., and Crushev, C. (2021). A Survey on Locality Sensitive Hashing Algorithms and Their Applications. arXiv.
13. Li, Y., Gu, C., Dullien, T., Vinyals, O., and Kohli, P. (2019). Graph Matching Networks for Learning the Similarity of Graph Structured Objects. arXiv.
14. Xu, X., Liu, C., Feng, Q., Yin, H., Song, L., and Song, D. (November, January 30). Neural Network-Based Graph Embedding for Cross-Platform Binary Code Similarity Detection. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, USA.10.1145/3133956.3134018 ↗
15. Mary, A., and Edison, A. (2023, January 19–21). Deep Fake Detection Using Deep Learning Techniques: A Literature Review. Proceedings of the 2023 International Conference on Control, Communication and Computing (ICCC), Thiruvananthapuram, India.10.1109/ICCC57789.2023.10164881 ↗
16. Deepfake Detection: A Systematic Literature Review10.1109/ACCESS.2022.3154404 ↗
17. Hassen, M., and Chan, P. (2017, January 22–24). Scalable Function Call Graph-Based Malware Classification. Proceedings of the Seventh ACM Conference on Data and Application Security and Privacy, Scottsdale, AZ, USA.10.1145/3029806.3029824 ↗
18. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. (2018). Graph Attention Networks. arXiv.
19. Hu, X., Chiueh, T., and Shin, K. (2009, January 9–13). Large-Scale Malware Indexing Using Function-Call Graphs. Proceedings of the 16th ACM Conference on Computer and Communications Security 2009, Chicago, IL, USA.10.1145/1653662.1653736 ↗
20. A Survey of Graph Edit Distance10.1007/s10044-008-0141-y ↗
21. A Similarity Metric Method of Obfuscated Malware Using Function-Call Graph10.1007/s11416-012-0175-y ↗
22. A Graph-Based Model for Malware Detection and Classification Using System-Call Groups10.1007/s11416-016-0267-1 ↗
23. Jiang, H., Turki, T., and Wang, J. (2018, January 17–20). Malware Detection Using Deep Learning and Graph Embedding. Proceedings of the 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, USA.10.1109/ICMLA.2018.00168 ↗
24. Grover, A., and Leskovec, J. (2016, January 13–17). Node2vec: Scalable Feature Learning for Networks. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939754 ↗
25. Busch, J., Kocheturov, A., Tresp, V., and Seidl, T. (2021, January 6–7). NF-GNN: Network Flow Graph Neural Networks for Malware Detection and Classification. Proceedings of the 33rd International Conference on Scientific and Statistical Database Management, Tampa, FL, USA.10.1145/3468791.3468814 ↗
26. Android Malware Detection Method Based on Graph Attention Networks and Deep Fusion of Multimodal Features10.1016/j.eswa.2023.121617 ↗
27. Behavior-Based Detection and Classification of Malicious Software Utilizing Structural Characteristics of Group Sequence Graphs10.1007/s11416-022-00423-4 ↗
28. On the Consistency of GNN Explanations for Malware Detection10.1016/j.ins.2025.122603 ↗
29. DeepCatra: Learning Flow- and Graph-Based Behaviours for Android Malware Detection10.1049/ise2.12082 ↗
30. Malware Detection with Dynamic Evolving Graph Convolutional Networks10.1002/int.22880 ↗
31. A Novel Malware Detection Method Based on Audit Logs and Graph Neural Network10.1016/j.engappai.2025.110524 ↗
32. A Resilience Recovery Method for Complex Traffic Network Security Based on Trend Forecasting10.1155/int/3715086 ↗
33. Kipf, T.N., and Welling, M. (2016). Semi-Supervised Classification with Graph Convolutional Networks. arXiv.
34. Wu, F., Zhang, T., de Souza, A.H., Fifty, C., Yu, T., and Weinberger, K.Q. (2019). Simplifying Graph Convolutional Networks. arXiv.
35. Oliveira, A. (2019). Malware Analysis Datasets: API Call Sequences. TechRxiv, Available online: https://www.kaggle.com/datasets/ang3loliveira/malware-analysis-datasets-api-call-sequences.
36. Li, Q., Han, Z., and Wu, X.-M. (2018). Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning. arXiv.10.1609/aaai.v32i1.11604 ↗
37. Oono, K., and Suzuki, T. (2021). Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. arXiv.