An Empirical Study on Software Defect Prediction Using CodeBERT Model
Source abstract
Abstract
Deep learning-based software defect prediction has been popular these days. Recently, the publishing of the CodeBERT model has made it possible to perform many software engineering tasks. We propose various CodeBERT models targeting software defect prediction, including CodeBERT-NT, CodeBERT-PS, CodeBERT-PK, and CodeBERT-PT. We perform empirical studies using such models in cross-version and cross-project software defect prediction to investigate if using a neural language model like CodeBERT could improve prediction performance. We also investigate the effects of different prediction patterns in software defect prediction using CodeBERT models. The empirical results are further discussed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Engineering Research · Software Reliability and Analysis Research · Software System Performance and Reliability
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
The Key Laboratory on Reliability and Environmental Engineering Technology, Beihang University, Beijing 100191, China
School of Reliability and Systems Engineering, Beihang University, Beijing 100191, China
References (50)
Show source-provided references
1. Defect prediction from static code features: Current results, limitations, new approaches10.1007/s10515-010-0069-5 ↗
2. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA.
3. Deep learning10.1038/nature14539 ↗
4. Wang, S., Liu, T., and Tan, L. (2016, January 14–22). Automatically learning semantic features for defect prediction. Proceedings of the 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), Austin, TX, USA.10.1145/2884781.2884804 ↗
5. Li, J., He, P., Zhu, J., and Lyu, M.R. (2017, January 25–29). Software defect prediction via convolutional neural network. Proceedings of the 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS), Prague, Czech Republic.10.1109/QRS.2017.42 ↗
6. A suitable ast node granularity and multi-kernel transfer convolutional neural network for cross-project defect prediction10.1109/ACCESS.2020.2985780 ↗
7. Pan, C., Lu, M., Xu, B., and Gao, H. (2019). An improved cnn model for within-project software defect prediction. Appl. Sci., 9.10.3390/app9102138 ↗
8. Software defect prediction via transformer10.1109/ITNEC48623.2020.9084745 ↗
9. Pathpair2vec: An ast path pair-based code representation method for defect prediction10.1016/j.cola.2020.100979 ↗
10. Hoang, T., Dam, H.K., Kamei, Y., Lo, D., and Ubayashi, N. (2019, January 25–31). Deepjit: An end-to-end deep learning framework for just-in-time defect prediction. Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), Montreal, QC, Canada.10.1109/MSR.2019.00016 ↗
11. Deepcpdp: Deep learning based cross-project defect prediction10.1109/ACCESS.2019.2961129 ↗
12. Seml: A semantic lstm model for software defect prediction10.1109/ACCESS.2019.2925313 ↗
13. Deep learning based software defect prediction10.1016/j.neucom.2019.11.067 ↗
14. SLDeep: Statement-level software defect prediction using deep-learning model on static code features10.1016/j.eswa.2019.113156 ↗
15. Hasanpour, A., Farzi, P., Tehrani, A., and Akbari, R. (2020). Software Defect Prediction Based on Deep Learning Models: Performance Study. arXiv.
16. Software defect prediction via LSTM10.1049/iet-sen.2019.0149 ↗
17. MPT-embedding: An unsupervised representation learning of code for software defect prediction10.1002/smr.2330 ↗
18. Semantic Feature Learning via Dual Sequences for Defect Prediction10.1109/ACCESS.2021.3051957 ↗
19. Meilong, S., He, P., Xiao, H., Li, H., and Zeng, C. (2020). An Approach to Semantic and Structural Features Learning for Software Defect Prediction. Math. Probl. Eng., 1–13.10.1155/2020/6038619 ↗
20. Omri, S., and Sinz, C. (2020, January 16–24). Deep Learning for Software Defect Prediction: A Survey. Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, Seoul, Korea.10.1145/3387940.3391463 ↗
21. Tian, J., and Tian, Y. (2020, January 3–6). A Model Based on Program Slice and Deep Learning for Software Defect Prediction. Proceedings of the 2020 29th International Conference on Computer Communications and Networks (ICCCN), Honolulu, HI, USA.10.1109/ICCCN49398.2020.9209658 ↗
22. Lin, X., Yang, J., and Li, Z. (2020, January 18–22). Software Defect Prediction with Spiking Neural Networks. Proceedings of the International Conference on Neural Information Processing, Bangkok, Thailand.10.1007/978-3-030-63823-8_75 ↗
23. Software defect prediction based on non-linear manifold learning and hybrid deep learning techniques
24. Wongpheng, K., and Visutsak, P. (2020, January 3–6). Software Defect Prediction using Convolutional Neural Network. Proceedings of the 2020 35th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), Nagoya, Japan.
25. An adversarial discriminative convolutional neural network for cross-project defect prediction10.1109/ACCESS.2020.2981869 ↗
26. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv.
27. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
28. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
29. Allamanis, M., Barr, E.T., Bird, C., and Sutton, C. (September, January 30). Suggesting accurate method and class names. Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy.10.1145/2786805.2786849 ↗
30. Mou, L., Li, G., Zhang, L., Wang, T., and Jin, Z. (2016, January 12–17). Convolutional neural networks over tree structures for programming language processing. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, Phoenix, AZ, USA.10.1609/aaai.v30i1.10139 ↗
31. Zhang, J., Wang, X., Zhang, H., Sun, H., Wang, K., and Liu, X. (2019, January 25–31). A novel neural source code representation based on abstract syntax tree. Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), Montreal, QC, Canada.10.1109/ICSE.2019.00086 ↗
32. Code2vec: Learning distributed representations of code
33. Li, Y., Tarlow, D., Brockschmidt, M., and Zemel, R.S. (2016, January 2–4). Gated graph sequence neural networks. Proceedings of the 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico.
34. Allamanis, M. (2019). The adverse effects of code duplication in machine learning models of code. Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Association for Computing Machinery.10.1145/3359591.3359735 ↗
35. Hellendoorn, V.J., Sutton, C., Singh, R., Maniatis, P., and Bieber, D. (2020, January 26–30). Global relational models of source code. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia.
36. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., and Jiang, D. (2020). CodeBERT: A pre-trained model for programming and natural languages. arXiv.10.18653/v1/2020.findings-emnlp.139 ↗
37. Yang, X., Lo, D., Xia, X., Zhang, Y., and Sun, J. (2015, January 3–5). Deep learning for just-in-time defect prediction. Proceedings of the 2015 IEEE International Conference on Software Quality, Reliability and Security, Vancouver, BC, Canada.10.1109/QRS.2015.14 ↗
38. Software defect prediction using stacked denoising autoencoders and two-stage ensemble learning10.1016/j.infsof.2017.11.008 ↗
39. Cost-Sensitive and Sparse Ladder Network for Software Defect Prediction10.1587/transinf.2019EDL8198 ↗
40. Torrey, L., and Shavlik, J. (2010). Transfer learning. Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, IGI Global.10.4018/978-1-60566-766-9.ch011 ↗
41. Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., and Liu, C. (2018, January 4–7). A survey on deep transfer learning. Proceedings of the International Conference on Artificial Neural Networks, Rhodes, Greece.10.1007/978-3-030-01424-7_27 ↗
42. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language models are few-shot learners. arXiv.
43. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
44. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv.
45. Raym, B., Hellendoorn, V., Godhane, S., Tu, Z., Bacchelli, A., and Devanbu, P. (2016, January 14–22). On the “naturalness” of buggy code. Proceedings of the 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), Austin, TX, USA.10.1145/2884781.2884848 ↗
46. A survey of machine learning for big code and naturalness10.1145/3212695 ↗
47. Buratti, L., Pujar, S., Bornea, M., McCarley, S., Zheng, Y., Rossiello, G., and Domeniconi, G. (2020). Exploring Software Naturalness through Neural Language Models. arXiv.
48. (2021, March 22). CodeBERT on HuggingFace. Available online: https://huggingface.co/microsoft/codebert-base.
49. Jureczko, M., and Madeyski, L. (2010, January 18–20). Towards identifying software project clusters with regard to defect prediction. Proceedings of the 6th International Conference on Predictive, Jinan, China.10.1145/1868328.1868342 ↗
50. Xu, J., Yan, L., Wang, F., and Ai, J. (2020, January 3–6). A GitHub-Based Data Collection Method for Software Defect Prediction. Proceedings of the 2019 6th International Conference on Dependable Systems and Their Applications (DSA), Harbin, China.10.1109/DSA.2019.00020 ↗