Natural Language Processing Application on Commit Messages: A Case Study on HEP Software
Source abstract
Abstract
Version Control and Source Code Management Systems, such as GitHub, contain a large amount of unstructured historical information of software projects. Recent studies have introduced Natural Language Processing (NLP) to help software engineers retrieve information from a very large collection of unstructured data. In this study, we have extended our previous study by increasing our datasets and machine learning and clustering techniques. We have followed a complex methodology made up of various steps. Starting from the raw commit messages we have employed NLP techniques to build a structured database. We have extracted their main features and used them as input of different clustering algorithms. Once each entry was labelled, we applied supervised machine learning techniques to build a prediction and classification model. We have developed a machine learning-based model to automatically classify commit messages of a software project. Our model exploits a ground-truth dataset that includes commit messages obtained from various GitHub projects belonging to the High Energy Physics context. The contribution of this paper is two-fold: it proposes a ground-truth database and it provides a machine learning prediction model that automatically identifies the more change-prone areas of code. Our model has obtained a very high average accuracy (0.9590), precision (0.9448), recall (0.9382), and F1-score (0.9360).
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Engineering Research · Scientific Computing and Data Management · Web Data Mining and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Statistical Sciences, University of Bologna, 40126 Bologna, Italy
INFN CNAF, 40126 Bologna, Italy
References (63)
Show source-provided references
1. Constantinou, E., and Kapitsaki, G.M. (September, January 31). Identifying Developers’ Expertise in Social Coding Platforms. Proceedings of the 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Limassol, Cyprus.10.1109/SEAA.2016.18 ↗
2. Thung, F., Bissyandé, T.F., Lo, D., and Jiang, L. (2013, January 5–8). Network Structure of Social Coding in GitHub. Proceedings of the 2013 17th European Conference on Software Maintenance and Reengineering, Genova, Italy.10.1109/CSMR.2013.41 ↗
3. Sarwar, M.U., Zafar, S., Mkaouer, M.W., Walia, G.S., and Malik, M.Z. (2020, January 12–15). Multi-label Classification of Commit Messages using Transfer Learning. Proceedings of the 2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), Coimbra, Portugal.10.1109/ISSREW51248.2020.00034 ↗
4. Bavota, G. (2016, January 14–18). Mining unstructured data in software repositories: Current & future trends. Proceedings of the 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering, SANER 2016, Osaka, Japan,.10.1109/SANER.2016.47 ↗
5. Jiang, S., Armaly, A., and McMillan, C. (November, January 30). Automatically generating commit messages from diffs using neural machine translation. Proceedings of the 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), Urbana, IL, USA.10.1109/ASE.2017.8115626 ↗
6. Jalote, P. (2005). An Integrated Approach to Software Engineering, Springer. Texts in Computer Science.
7. Integrating Natural Language Processing and Software Engineering
8. Venigalla, A.S.M., and Chimalakonda, S. (2021, January 25–28). Understanding Emotions of Developer Community towards Software Documentation. Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS), Madrid, Spain.10.1109/ICSE-SEIS52602.2021.00018 ↗
9. NLP-assisted software testing: A systematic mapping of the literature10.1016/j.infsof.2020.106321 ↗
10. Siow, J., Gao, C., Fan, L., Chen, S., and Liu, Y. (2020, January 18–21). CORE: Automating Review Recommendation for Code Changes. Proceedings of the 27th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), London, ON, Canada.10.1109/SANER48275.2020.9054794 ↗
11. Zhao, L., Alhoshan, W., Ferrari, A., Letsholo, K.J., Ajagbe, M.A., Chioasca, E.V., and Batista-Navarro, R.T. (2020). Natural Language Processing (NLP) for Requirements Engineering: A Systematic Mapping Study. arXiv.10.1145/3444689 ↗
12. Mapping Bug Reports to Relevant Files: A Ranking Model, a Fine-Grained Benchmark, and Feature Evaluation10.1109/TSE.2015.2479232 ↗
13. Gilson, F., and Weyns, D. (2019, January 25–26). When Natural Language Processing Jumps into Collaborative Software Engineering. Proceedings of the 2019 IEEE International Conference on Software Architecture Companion (ICSA-C), Hamburg, Germany.10.1109/ICSA-C.2019.00049 ↗
14. Catolino, G., and Ferrucci, F. (2018, January 20). Ensemble techniques for software change prediction: A preliminary investigation. Proceedings of the 2018 IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation (MaLTeSQuE), Campobasso, Italy.10.1109/MALTESQUE.2018.8368455 ↗
15. Enhancing change prediction models using developer-related factors10.1016/j.jss.2018.05.003 ↗
16. Examining the Potentially Confounding Effect of Class Size on the Associations between Object-Oriented Metrics and Change-Proneness10.1109/TSE.2009.32 ↗
17. Assessment of Code Smell for Predicting Class Change Proneness Using Machine Learning10.1109/ACCESS.2019.2905133 ↗
18. Ronchieri, E., Yang, Y., Canaparo, M., Costantini, A., Duma, D.C., and Salomoni, D. (November, January 31). A new code change prediction dataset: A case study based on HEP software. Proceedings of the IEEE NSS MIC 2020, Boston, MA, USA.10.1109/NSS/MIC42677.2020.9508053 ↗
19. Ronchieri, E., Canaparo, M., and Yang, Y. (2021, January 22–26). Using Natural Language Processing to Extract Information from Unstructured code-change version control data: Lessons learned. Proceedings of the International Symposium on Grids & Clouds, Taipei, Taiwan.10.22323/1.378.0025 ↗
20. Customer satisfaction and natural language processing10.1016/j.jbusres.2020.11.065 ↗
21. Ozturkmenoglu, O., and Alpkocak, A. (2012, January 2–4). Comparison of different lemmatization approaches for information retrieval on Turkish text collection. Proceedings of the 2012 International Symposium on Innovations in Intelligent Systems and Applications, Trabzon, Turkey.10.1109/INISTA.2012.6246934 ↗
22. Patil, L.H., and Atique, M. (2012, January 22–23). A Novel Approach for Feature Selection Method TF- IDF in Document Clustering. Proceedings of the 2013 3rd IEEE International Advance Computing Conference (IACC), Ghaziabad, India.10.1109/IAdCC.2013.6514339 ↗
23. dos Santos, G.E., and Figueiredo, E. (2020, January 9–13). Commit Classification using Natural Language Processing: Experiments over Labeled Datasets. Proceedings of the CIbSE, Curitiba, Brazil.
24. Mining commit messages to enhance software refactorings recommendation: A machine learning approach
25. Sagar, P.S., AlOmar, E.A., Mkaouer, M.W., Ouni, A., and Newman, C.D. (2021). Comparing Commit Messages and Source Code Metrics for the Prediction Refactoring Activities. Algorithms, 14.10.3390/a14100289 ↗
26. Recommending refactorings via commit message analysis10.1016/j.infsof.2020.106332 ↗
27. Predicting technical debt from commit contents: Reproduction and extension with automated feature selection10.1007/s11219-020-09520-3 ↗
28. Jung, T. (2021). CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model. arXiv.10.18653/v1/2021.nlp4prog-1.3 ↗
29. Automated change-prone class prediction on unlabeled dataset using unsupervised method10.1016/j.infsof.2017.07.003 ↗
30. Levin, S., Aviv, T., and Aviv, T. (2017). Boosting Automatic Commit Classification Into Maintenance Activities By Utilizing Source Code Changes. arXiv.10.1145/3127005.3127016 ↗
31. Messaoud, M.B. (2019, January 8–12). On the Classification of Software Change Messages using Multi-label Active Learning. Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, Limassol, Cyprus.
32. Barnett, J.G., Gathuru, C.K., Soldano, L.S., and McIntosh, S. (2016, January 14–22). The relationship between commit message detail and defect proneness in Java projects on GitHub. Proceedings of the 13th Working Conference on Mining Software Repositories, MSR 2016, Austin, TX, USA.10.1145/2901739.2903496 ↗
33. Levin, S., and Yehudai, A. (2017, January 8). Boosting Automatic Commit Classification Into Maintenance Activities By Utilizing Source Code Changes. Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering, Toronto, ON, Canada.10.1145/3127005.3127016 ↗
34. Zhong, S., Khoshgoftaar, T.M., and Seliya, N. (2004, January 25–26). Unsupervised learning for expert-based software quality estimation. Proceedings of the Eighth IEEE Internation Symposium on High Assurance Systems Engineering, Tampa, FL, USA.
35. On the nature of the nature of law10.25162/arsp-2012-0036 ↗
36. Yamauchi, K., Yang, J., Hotta, K., Higo, Y., and Kusumoto, S. (October, January 29). Clustering commits for understanding the intents of implementation. Proceedings of the 30th International Conference on Software Maintenance and Evolution, ICSME 2014, Victoria, BC, Canada.10.1109/ICSME.2014.63 ↗
37. Product selection based on sentiment analysis of online reviews: An intuitionistic fuzzy TODIM method10.1007/s40747-022-00678-w ↗
38. Zhou, L., Tang, L., and Zhang, Z. (2022). Extracting and ranking product features in consumer reviews based on evidence theory. J. Ambient. Intell. Humaniz. Comput., 1868–5145.10.1007/s12652-021-03664-1 ↗
39. Attention-based BiLSTM models for personality recognition from user-generated content10.1016/j.ins.2022.03.038 ↗
40. Characteristic Analysis of Judgment Debtors Based on Hesitant Fuzzy Linguistic Clustering Method10.1109/ACCESS.2021.3107604 ↗
41. Text Classification Using Novel Term Weighting Scheme-Based Improved TF-IDF for Internet Media Reports
42. Ali, A., Bin Faheem, Z., Waseem, M., Draz, U., Safdar, Z., Hussain, S., and Yaseen, S. (2020, January 5–7). Systematic Review: A State of Art ML Based Clustering Algorithms for Data Mining. Proceedings of the 2020 IEEE 23rd International Multitopic Conference (INMIC), Bahawalpur, Pakistan.10.1109/INMIC50486.2020.9318060 ↗
43. Kapil, S., Chawla, M., and Ansari, M.D. (2016, January 22–24). On K-means data clustering algorithm with genetic algorithm. Proceedings of the 2016 Fourth International Conference on Parallel, Distributed and Grid Computing (PDGC), Waknaghat, India.10.1109/PDGC.2016.7913145 ↗
44. Software Module Clustering: An In-Depth Literature Analysis
45. Nainggolan, R., Perangin-Angin, R., Simarmata, E., and Tarigan, A.F. (2019). Improved the Performance of the K-Means Cluster Using the Sum of Squared Error (SSE) optimized by using the Elbow Method. J. Phys. Conf. Ser., 1361.10.1088/1742-6596/1361/1/012015 ↗
46. Research on K-Value Selection Method of K-Means Clustering Algorithm
47. The silhouette width criterion for clustering and association mining to select image features10.18178/ijmlc.2018.8.1.665 ↗
48. Kogan, J., Nicholas, C., and Teboulle, M. (2006). A Survey of Clustering Data Mining Techniques. Grouping Multidimensional Data: Recent Advances in Clustering, Springer.10.1007/3-540-28349-8 ↗
49. Daniel Jurafsky, J.H.M. (2008). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, PEARSON.
50. Introduction to Machine Learning
51. Classification and Regression Trees, Bagging, and Boosting10.1016/S0169-7161(04)24011-1 ↗
52. Cross-Validation Methods10.1006/jmps.1999.1279 ↗
53. Classification of heart sound signals using a novel deep WaveNet model10.1016/j.cmpb.2020.105604 ↗
54. (2022, September 15). ALISW. Available online: https://github.com/alisw.
55. (2022, September 15). LHCB. Available online: https://github.com/lhcb.
56. (2022, September 15). CMS-SW. Available online: https://github.com/cms-sw.
57. (2022, September 15). ROOT. Available online: https://github.com/root-project/root.
58. Swanson, E.B. (1976, January 13–15). The dimensions of maintenance. Proceedings of the 2nd International Conference on Software Engineering, San Francisco, CA, USA.
59. Hindle, A., German, D.M., and Holt, R. (2018, January 10–11). What Do Large Commits Tell Us? A Taxonomical Study of Large Commits. Proceedings of the 2008 International Working Conference on Mining Software Repositories, MSR ’08, Leipzig, Germany.10.1145/1370750.1370773 ↗
60. Early prediction for merged vs abandoned code changes in modern code reviews10.1016/j.infsof.2021.106756 ↗
61. A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments10.1016/j.jss.2021.110911 ↗
62. Khatiwada, S., Kelly, M., and Mahmoud, A. (2016, January 16–17). STAC: A tool for Static Textual Analysis of Code. Proceedings of the 2016 IEEE 24th International Conference on Program Comprehension (ICPC), Austin, TX, USA.10.1109/ICPC.2016.7503746 ↗
63. Supervised and Traditional Term Weighting Methods for Automatic Text Categorization10.1109/TPAMI.2008.110 ↗