Structure-Enhanced Prompt Learning for Graph-Based Code Vulnerability Detection
Source abstract
Abstract
Recent advances in prompt learning have opened new avenues for enhancing natural language understanding in domain-specific tasks, including code vulnerability detection. Motivated by the limitations of conventional binary classification methods in capturing complex code semantics, we propose a novel framework that integrates a two-stage prompt optimization mechanism with hierarchical representation learning. Our approach leverages graphon theory to generate task-adaptive, structurally enriched prompts by encoding both contextual and graphical information into trainable vector representations. To further enhance representational capacity, we incorporate the pretrained model CodeBERTScore, a syntax-aware encoder, and Graph Neural Networks, enabling comprehensive modeling of both local syntactic features and global structural dependencies. Experimental results on three public datasets—FFmpeg+Qemu, SVulD and Reveal—demonstrate that our method performs competitively across all benchmarks, achieving accuracy rates of 64.40%, 83.44% and 90.69%, respectively. These results underscore the effectiveness of combining prompt-based learning with graph-based structural modeling, offering a more accurate and robust solution for automated vulnerability detection.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Reliability and Analysis Research · Software Engineering Research · Software Testing and Debugging Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Cybersapce Security, Hainan University, Haikou 570228, China
School of Computer Science and Technology, Hainan University, Haikou 570228, China
Funding
National Natural Science Foundation of China · National Key Research and Development Program of China · Key Research and Development Program of Hainan Province · Hainan Province Intelligent Software Engineering Research Center · Key Laboratory of Big Data and Smart Services of Hainan Province
References (55)
Show source-provided references
1. MITRE (2025, January 15). Common Vulnerabilities and Exposures. Available online: https://www.cve.org/.
2. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020, January 6–12). Language models are few-shot learners. Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA.
3. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual.
4. Hu, S., Ding, N., Wang, H., Liu, Z., Wang, J., Li, J., Wu, W., and Sun, M. (2022, January 22–27). Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland.10.18653/v1/2022.acl-long.158 ↗
5. Chen, X., Zhang, N., Xie, X., Deng, S., Yao, Y., Tan, C., Huang, F., Si, L., and Chen, H. (2022, January 25–29). Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. Proceedings of the ACM Web Conference 2022, Lyon, France.10.1145/3485447.3511998 ↗
6. Alon, U., and Yahav, E. (2021). On the Bottleneck of Graph Neural Networks and its Practical Implications. arXiv.
7. Wen, X.C., Chen, Y., Gao, C., Zhang, H., Zhang, J.M., and Liao, Q. (2023, January 14–20). Vulnerability Detection with Graph Simplification and Enhanced Graph Representation Learning. Proceedings of the 45th International Conference on Software Engineering, ICSE’23, Melbourne, Australia.10.1109/ICSE48619.2023.00191 ↗
8. Deepwukong: Statically detecting software vulnerabilities using deep graph neural network10.1145/3436877 ↗
9. Hin, D., Kan, A., Chen, H., and Babar, M.A. (2022, January 23–24). LineVD: Statement-level Vulnerability Detection using Graph Neural Networks. Proceedings of the 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR), Pittsburgh, PA, USA.10.1145/3524842.3527949 ↗
10. Diaconis, P., and Janson, S. (2007). Graph limits and exchangeable random graphs. arXiv.
11. Kim, Y. (2014, January 25–29). Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1181 ↗
12. Zhou, Y., Liu, S., Siow, J., Du, X., and Liu, Y. (2019). Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks. arXiv.
13. Ni, C., Yin, X., Yang, K., Zhao, D., Xing, Z., and Xia, X. (2023, January 5–7). Distinguishing Look-Alike Innocent and Vulnerable Code by Subtle Semantic Representation Learning and Explanation. Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA.10.1145/3611643.3616358 ↗
14. Deep learning based vulnerability detection: Are we there yet?10.1109/TSE.2021.3087402 ↗
15. Russell, R., Kim, L., Hamilton, L., Lazovich, T., Harer, J., Ozdemir, O., Ellingwood, P., and McConley, M. (2018, January 17–20). Automated Vulnerability Detection in Source Code Using Deep Representation Learning. Proceedings of the 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, USA.10.1109/ICMLA.2018.00120 ↗
16. Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., Deng, Z., and Zhong, Y. (2018, January 18–21). VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. Proceedings of the 2018 Network and Distributed System Security Symposium, San Diego, CA, USA.10.14722/ndss.2018.23158 ↗
17. SySeVR: A framework for using deep learning to detect software vulnerabilities10.1109/TDSC.2021.3051525 ↗
18. Peng, B., Liu, Z., Zhang, J., and Su, P. (2023, January 18–23). CEVulDet: A Code Edge Representation Learnable Vulnerability Detector. Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN), Gold Coast, Australia.10.1109/IJCNN54540.2023.10191361 ↗
19. CPVD: Cross project vulnerability detection based on graph attention network and domain adaptation10.1109/TSE.2023.3285910 ↗
20. Meta-path based attentional graph learning model for vulnerability detection10.1109/TSE.2023.3340267 ↗
21. Graph Confident Learning for Software Vulnerability Detection10.1016/j.engappai.2024.108296 ↗
22. Enhancing vulnerability detection via AST decomposition and neural sub-tree encoding10.1016/j.eswa.2023.121865 ↗
23. Wen, X.C., Gao, C., Gao, S., Xiao, Y., and Lyu, M.R. (2024, January 16–20). SCALE: Constructing Structured Natural Language Comment Trees for Software Vulnerability Detection. Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria.10.1145/3650212.3652124 ↗
24. Wu, Y., Zou, D., Dou, S., Yang, W., Xu, D., and Jin, H. (2022, January 22–24). VulCNN: An image-inspired scalable vulnerability detection system. Proceedings of the 44th International Conference on Software Engineering, Pittsburgh, PA, USA.10.1145/3510003.3510229 ↗
25. Buratti, L., Pujar, S., Bornea, M., McCarley, S., Zheng, Y., Rossiello, G., Morari, A., Laredo, J., Thost, V., and Zhuang, Y. (2020). Exploring Software Naturalness through Neural Language Models. arXiv.
26. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., and Jiang, D. (2020, January 16–20). CodeBERT: A Pre-Trained Model for Programming and Natural Languages. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020, Online.10.18653/v1/2020.findings-emnlp.139 ↗
27. Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., and Fu, S. (2021). GraphCodeBERT: Pre-training Code Representations with Data Flow. arXiv.
28. Zhou, S., Alon, U., Agarwal, S., and Neubig, G. (2023, January 6–10). CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.10.18653/v1/2023.emnlp-main.859 ↗
29. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
30. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H.P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., and Brockman, G. (2021). Evaluating Large Language Models Trained on Code. arXiv.
31. Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A.H., and Riedel, S. (2019, January 3–7). Language models as knowledge bases?. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1250 ↗
32. How can we know what language models know?10.1162/tacl_a_00324 ↗
33. Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., and Singh, S. (2020, January 16–20). Autoprompt: Eliciting knowledge from language models with automatically generated prompts. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.346 ↗
34. Lester, B., Al-Rfou, R., and Constant, N. (2021, January 7–11). The power of scale for parameter-efficient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.243 ↗
35. Zhong, Z., Friedman, D., and Chen, D. (2021, January 6–11). Factual probing is [mask]: Learning vs. learning to recall. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.398 ↗
36. Li, Y., Liang, F., Zhao, L., Cui, Y., Ouyang, W., Shao, J., Yu, F., and Yan, J. (2022). Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm. arXiv.
37. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing
38. Prompt Tuning in Code Intelligence: An Experimental Evaluation10.1109/TSE.2023.3313881 ↗
39. Zhang, C., Liu, H., Zeng, J., Yang, K., Li, Y., and Li, H. (2024, January 14–20). Prompt-Enhanced Software Vulnerability Detection Using ChatGPT. Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, Lisbon, Portugal.10.1145/3639478.3643065 ↗
40. Learning to prompt for vision-language models10.1007/s11263-022-01653-1 ↗
41. Yamaguchi, F., Golde, N., Arp, D., and Rieck, K. (2014, January 18–21). Modeling and discovering vulnerabilities with code property graphs. Proceedings of the 2014 IEEE Symposium on Security and Privacy, Berkeley, CA, USA.10.1109/SP.2014.44 ↗
42. Combining graph-based learning with automated data collection for code vulnerability detection10.1109/TIFS.2020.3044773 ↗
43. Zhou, K., Yang, J., Loy, C.C., and Liu, Z. (2022, January 18–24). Conditional prompt learning for vision-language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01631 ↗
44. A survey of statistical network models10.1561/2200000005 ↗
45. Lovász, L. (2012). Large Networks and Graph Limits, American Mathematical Society.10.1090/coll/060 ↗
46. Quick approximation to matrices and applications10.1007/s004930050052 ↗
47. Matrix estimation by universal singular value thresholding10.1214/14-AOS1272 ↗
48. Fan, J., Li, Y., Wang, S., and Nguyen, T.N. (2020, January 29–30). A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. Proceedings of the 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR), Seoul, Republic of Korea.10.1145/3379597.3387501 ↗
49. Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., and Tang, D. (2021). Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR.
50. Wen, X.C., Wang, X., Gao, C., Wang, S., Liu, Y., and Gu, Z. (2023, January 11–15). When Less is Enough: Positive and Unlabeled Learning Model for Vulnerability Detection. Proceedings of the 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), Luxembourg.10.1109/ASE56229.2023.00144 ↗
51. Wang, Y., Wang, W., Joty, S., and Hoi, S.C. (2021, January 7–11). CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.685 ↗
52. Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., and Yin, J. (2022, January 22–27). Unixcoder: Unified cross-modal pre-training for code representation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland.10.18653/v1/2022.acl-long.499 ↗
53. Fu, M., and Tantithamthavorn, C. (2022, January 23–24). LineVul: A Transformer-based Line-Level Vulnerability Prediction. Proceedings of the 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR), Pittsburgh, PA, USA.10.1145/3524842.3528452 ↗
54. Vulnerability Detection by Learning From Syntax-Based Execution Paths of Code10.1109/TSE.2023.3286586 ↗
55. McInnes, L., Healy, J., and Melville, J. (2020). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv.