An Improved CNN Model for Within-Project Software Defect Prediction
Source abstract
Abstract
To improve software reliability, software defect prediction is used to find software bugs and prioritize testing efforts. Recently, some researchers introduced deep learning models, such as the deep belief network (DBN) and the state-of-the-art convolutional neural network (CNN), and used automatically generated features extracted from abstract syntax trees (ASTs) and deep learning models to improve defect prediction performance. However, the research on the CNN model failed to reveal clear conclusions due to its limited dataset size, insufficiently repeated experiments, and outdated baseline selection. To solve these problems, we built the PROMISE Source Code (PSC) dataset to enlarge the original dataset in the CNN research, which we named the Simplified PROMISE Source Code (SPSC) dataset. Then, we proposed an improved CNN model for within-project defect prediction (WPDP) and compared our results to existing CNN results and an empirical study. Our experiment was based on a 30-repetition holdout validation and a 10 * 10 cross-validation. Experimental results showed that our improved CNN model was comparable to the existing CNN model, and it outperformed the state-of-the-art machine learning models significantly for WPDP. Furthermore, we defined hyperparameter instability and examined the threat and opportunity it presents for deep learning models on defect prediction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Engineering Research · Software Reliability and Analysis Research · Software System Performance and Reliability
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
The Key Laboratory on Reliability and Environmental Engineering Technology, Beihang University, Beijing 100191, China
School of Reliability and System Engineering, Beihang University, Beijing 100191, China
References (64)
Show source-provided references
1. Defect prediction from static code features: Current results, limitations, new approaches10.1007/s10515-010-0069-5 ↗
2. Moser, R., Pedrycz, W., and Succi, G. (2008, January 15). A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction. Proceedings of the 30th International Conference on Software Engineering, Leipzig, Germany.10.1145/1368088.1368114 ↗
3. Tan, M., Tan, L., Dara, S., and Mayeux, C. (2015, January 16–24). Online Defect Prediction for Imbalanced Data. Proceedings of the 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering (ICSE), Florence, Italy.10.1109/ICSE.2015.139 ↗
4. Nam, J., Pan, S.J., and Kim, S. (2013, January 18–26). Transfer defect learning. Proceedings of the International Conference of Software Engineering, San Francisco, CA, USA.10.1109/ICSE.2013.6606584 ↗
5. Nam, J. (2014). Survey on Software Defect Prediction. [Ph.D. Thesis, The Hong Kong University of Science and Technology].
6. Lyu, M.R. (1996). Handbook of Software Reliability Engineering, IEEE Computer Society Press.
7. Halstead, M.H. (1977). Elements of Software Science, Elsevier Science Inc.
8. A Complexity Measure10.1109/TSE.1976.233837 ↗
9. A Metrics Suite for Object Oriented Design10.1109/32.295895 ↗
10. An evaluation of the MOOD set of object-oriented software metrics10.1109/32.689404 ↗
11. Jiang, T., Tan, L., and Kim, S. (2013, January 11–15). Personalized defect prediction. Proceedings of the Automated Software Engineering, Silicon Valley, CA, USA.10.1109/ASE.2013.6693087 ↗
12. Empirical validation of object-oriented metrics on open source software for fault prediction10.1109/TSE.2005.112 ↗
13. Empirical Analysis of Object-Oriented Design Metrics for Predicting High and Low Severity Faults10.1109/TSE.2006.102 ↗
14. Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings10.1109/TSE.2008.35 ↗
15. Data Mining Static Code Attributes to Learn Defect Predictors10.1109/TSE.2007.256941 ↗
16. Jing, X.Y., Ying, S., Wu, S.S., and Liu, J. (June, January 31). Dictionary learning based software defect prediction. Proceedings of the 36th International Conference on Software Engineering, Hyderabad, India.
17. Hindle, A., Barr, E.T., Su, Z., Gabel, M., and Devanbu, P. (2012, January 2–9). On the naturalness of software. Proceedings of the 2012 34th International Conference on Software Engineering (ICSE), Zurich, Switzerland.10.1109/ICSE.2012.6227135 ↗
18. Nguyen, A.T., and Nguyen, T.N. (2015, January 16–24). Graph-based statistical language model for code. Proceedings of the 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Florence, Italy.10.1109/ICSE.2015.336 ↗
19. Automatically identifying code features for software defect prediction: Using AST N-grams10.1016/j.infsof.2018.10.001 ↗
20. White, M., Vendome, C., Linares-Vasquez, M., and Poshyvanyk, D. (2015, January 16–17). Toward Deep Learning Software Repositories. Proceedings of the 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories (MSR), Florence, Italy.10.1109/MSR.2015.38 ↗
21. Machine translation-based bug localization technique for bridging lexical gap10.1016/j.infsof.2018.03.003 ↗
22. Tu, Z., Su, Z., and Devanbu, P. (2014, January 16–21). On the localness of software. Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, Hong Kong, China.10.1145/2635868.2635875 ↗
23. Wang, S., Liu, T., and Tan, L. (2016, January 14–22). Automatically learning semantic features for defect prediction. Proceedings of the 38th International Conference on Software, Austin, TX, USA.10.1145/2884781.2884804 ↗
24. Li, J., He, P., Zhu, J., and Lyu, M.R. (2017, January 25–29). Software Defect Prediction via Convolutional Neural Network. Proceedings of the 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS), Prague, Czech Republic.10.1109/QRS.2017.42 ↗
25. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–8). Imagenet classification with deep convolutional neural networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA.
26. Deep learning10.1038/nature14539 ↗
27. Arcuri, A., and Briand, L. (2011, January 21–28). A practical guide for using statistical tests to assess randomized algorithms in software engineering. Proceedings of the 2011 33rd International Conference on Software Engineering (ICSE), Honolulu, HI, USA.10.1145/1985793.1985795 ↗
28. An Empirical Comparison of Model Validation Techniques for Defect Prediction Models10.1109/TSE.2016.2584050 ↗
29. He, Z., Péters, F., Menzies, T., and Yang, Y. (2013, January 10–11). Learning from Open-Source Projects: An Empirical Study on Defect Prediction. Proceedings of the 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Baltimore, MD, USA.10.1109/ESEM.2013.20 ↗
30. Jureczko, M., and Madeyski, L. (2010, January 12–13). Towards identifying software project clusters with regard to defect prediction. Proceedings of the 6th International Conference on Predictive, Timişoara, Romania.10.1145/1868328.1868342 ↗
31. Herbold, S., Trautsch, A., and Grabowski, J. (2018). Correction of A Comparative Study to Benchmark Cross-project Defect Prediction Approaches. IEEE Trans. Softw. Eng., 1.
32. Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., and Penn, G. (2012, January 25–30). Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition. Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan.10.1109/ICASSP.2012.6288864 ↗
33. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
34. Zhang, X., Zhao, J., and LeCun, Y. (2015, January 7—12). Character-level Convolutional Networks for Text Classification. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
35. Sparse activity and sparse connectivity in supervised learning
36. Rong, X. (2019, January 24). word2vec Parameter Learning Explained. Available online: https://arxiv.org/abs/1411.2738.
37. Dropout: A simple way to prevent neural networks from overfitting
38. Peng, H., Mou, L., Li, G., Liu, Y., Zhang, L., and Jin, Z. (2015, January 7—11). Building Program Vector Representations for Deep Learning. Proceedings of the Image Analysis and Processing—ICIAP 2015, Genoa, Italy.
39. He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 11). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Araucano Park, Chile.10.1109/ICCV.2015.123 ↗
40. Glorot, X., and Bengio, Y. (2010, January 13–15). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy.
41. Bottou, L. (2010). Large-Scale Machine Learning with Stochastic Gradient Descent. Proceedings of the COMPSTAT’2010, Springer.10.1007/978-3-7908-2604-3_16 ↗
42. Kingma, D.P., and Ba, J. (2019, February 25). Adam: A Method for Stochastic Optimization. Available online: https://arxiv.org/abs/1412.6980.
43. D’Ambros, M., Lanza, M., and Robbes, R. (2010, January 2–3). An extensive comparison of bug prediction approaches. Proceedings of the 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010), Cape Town, South Africa.10.1109/MSR.2010.5463279 ↗
44. Zimmermann, T., Nagappan, N., Gall, H., Giger, E., and Murphy, B. (2009, January 24–28). Cross-project defect prediction: A large scale experiment on data vs. domain vs. process. Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, Amsterdam, The Netherlands.10.1145/1595696.1595713 ↗
45. Data Quality: Some Comments on the NASA Software Defect Datasets10.1109/TSE.2013.11 ↗
46. Herzig, K., Just, S., Rau, A., and Zeller, A. (2013, January 4–7). Predicting defects using change genealogies. Proceedings of the 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), Pasadena, CA, USA.10.1109/ISSRE.2013.6698911 ↗
47. Wu, R., Zhang, H., Kim, S., and Cheung, S.C. (2011, January 5–9). Relink: Recovering links between bugs and changes. Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, Szeged, Hungary.10.1145/2025113.2025120 ↗
48. The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance10.1080/01621459.1937.10503522 ↗
49. A simple sequentially rejective multiple test procedure
50. Yang, X., Lo, D., Xia, X., Zhang, Y., and Sun, J. (2015, January 3–5). Deep Learning for Just-in-Time Defect Prediction. Proceedings of the 2015 IEEE International Conference on Software Quality, Reliability and Security (QRS), Vancouver, BC, Canada.10.1109/QRS.2015.14 ↗
51. Software defect prediction using stacked denoising autoencoders and two-stage ensemble learning10.1016/j.infsof.2017.11.008 ↗
52. Using Cross-Entropy Value of Code for Better Defect Prediction
53. Dam, H.K., Pham, T., Ng, S.W., Tran, T., Grundy, J., Ghose, A., and Kim, C.J. (2019, January 24). A Deep Tree-Based Model for Software Defect Prediction. Available online: https://arxiv.org/abs/1802.00921.
54. Phan, A.V., Nguyen, L.M., and Bui, L.T. (2017, January 6–8). Convolutional neural networks over control flow graphs for software defect prediction. Proceedings of the 2017 IEEE 29th International Conference on Tools with Artificial Intelligence (ICTAI), Boston, MA, USA.
55. Phan, A.V., and Nguyen, L.M. (2017, January 15–17). Convolutional neural networks on assembly code for predicting software defects. Proceedings of the 2017 21st Asia Pacific Symposium on Intelligent and Evolutionary Systems (IES), Hanoi, Vietnam.10.1109/IESYS.2017.8233558 ↗
56. Cheng, J., Guo, J., and Cleland-Huang, J. (2017, January 20–28). Semantically Enhanced Software Traceability Using Deep Learning Techniques. Proceedings of the 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), Buenos Aires, Argentina.
57. Li, L., Feng, H., Zhuang, W., Meng, N., and Ryder, B. (2017, January 17–24). CCLearner: A Deep Learning-Based Clone Detection Approach. Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME), Shangai, China.10.1109/ICSME.2017.46 ↗
58. Lam, A.N., Nguyen, A.T., Nguyen, H.A., and Nguyen, T.N. (2017, January 22–23). Bug localization with combination of deep learning and information retrieval. Proceedings of the 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC), Buenos Aires, Argentina.10.1109/ICPC.2017.24 ↗
59. DeepBugs: A learning approach to name-based bug detection10.1145/3276517 ↗
60. Reyes, J., Ramirez, D., and Paciello, J. (2016, January 15–17). Automatic Classification of Source Code Archives by Programming Language: A Deep Learning Approach. Proceedings of the International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA.10.1109/CSCI.2016.0103 ↗
61. Zekany, S., Rings, D., Harada, N., Laurenzano, M.A., Tang, L., and Mars, J. (2016, January 15–19). CrystalBall: Statically analyzing runtime behavior via deep sequence learning. Proceedings of the 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Taipei, Taiwan.10.1109/MICRO.2016.7783727 ↗
62. Corley, C.S., Damevski, K., and Kraft, N.A. (October, January 29). Exploring the use of deep learning for feature location. Proceedings of the 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), Bremen, Germany.10.1109/ICSM.2015.7332513 ↗
63. Pang, Y., Xue, X., and Wang, H. (2017, January 2–4). Predicting Vulnerable Software Components through Deep Neural Network. Proceedings of the 2017 International Conference on Deep Learning Technologies, Chengdu China.10.1145/3094243.3094245 ↗
64. Bandara, U., and Wijayarathna, G. (2013, January 3–7). Deep Neural Networks for Source Code Author Identification. Proceedings of the 20th International Conference, Daegu, Korea.10.1007/978-3-642-42042-9_46 ↗