Evaluating the Usability and Functionality of Intelligent Source Code Completion Assistants: A Comprehensive Review
Source abstract
Abstract
As artificial intelligence advances, source code completion assistants are becoming more advanced and powerful. Existing traditional assistants are no longer up to all the developers’ challenges. Traditional assistants usually present proposals in alphabetically sorted lists, which does not make a developer’s tasks any easier (i.e., they still have to search and filter an appropriate proposal manually). As a possible solution to the presented issue, intelligent assistants that can classify suggestions according to relevance in particular contexts have emerged. Artificial intelligence methods have proven to be successful in solving such problems. Advanced intelligent assistants not only take into account the context of a particular source code but also, more importantly, examine other available projects in detail to extract possible patterns related to particular source code intentions. This is how intelligent assistants try to provide developers with relevant suggestions. By conducting a systematic literature review, we examined the current intelligent assistant landscape. Based on our review, we tested four intelligent assistants and compared them according to their functionality. GitHub Copilot, which stood out, allows suggestions in the form of complete source code sections. One would expect that intelligent assistants, with their outstanding functionalities, would be one of the most popular helpers in a developer’s toolbox. However, through a survey we conducted among practitioners, the results, surprisingly, contradicted this idea. Although intelligent assistants promise high usability, our questionnaires indicate that usability improvements are still needed. However, our research data show that experienced developers value intelligent assistants highly, highlighting their significant utility for the experienced developers group when compared to less experienced individuals. The unexpectedly low net promoter score (NPS) for intelligent code assistants in our study was quite surprising, highlighting a stark contrast between the anticipated impact of these advanced tools and their actual reception among developers.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Engineering Research · Software Engineering Techniques and Practices · Open Source Software Innovations
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Electrical Engineering and Computer Science, University of Maribor, 2000 Maribor, Slovenia
Funding
Republic of Slovenia, Ministry of Higher Education, Science and Innovation
References (47)
Show source-provided references
1. Hussain, Y., Huang, Z., Zhou, Y., and Wang, S. (2019). DeepVS: An Efficient and Generic Approach for Source Code Modeling Usage. arXiv.10.1049/el.2020.0500 ↗
2. Svyatkovskiy, A., Lee, S., Hadjitofi, A., Riechert, M., Franco, J., and Allamanis, M. (2020). Fast and Memory-Efficient Neural Code Completion. arXiv.10.1109/MSR52588.2021.00045 ↗
3. Survey of intelligent code completion
4. AI-Driven Development Is Here: Should You Worry?10.1109/MS.2021.3133805 ↗
5. GitHub Copilot (2022, October 24). GitHub Copilot Your AI Pair Programmer. Available online: https://copilot.github.com/.
6. Tabnine (2022, October 24). Tabnine Docs. Available online: https://www.tabnine.com/.
7. Hussain, Y., Huang, Z., Zhou, Y., and Wang, S. (2019). Deep Transfer Learning for Source Code Modeling. arXiv.
8. Hellendoorn, V.J., Proksch, S., Gall, H.C., and Bacchelli, A. (2019, January 25–31). When Code Completion Fails: A Case Study on Real-World Completions. Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), Montreal, QC, Canada.10.1109/ICSE.2019.00101 ↗
9. Aye, G.A., Kim, S., and Li, H. (2020). Learning Autocompletion from Real-World Datasets. arXiv.10.1109/ICSE-SEIP52600.2021.00022 ↗
10. Kitchenham, B.A. (2012, January 19–20). Systematic Review in Software Engineering: Where We Are and Where We Should Be Going. Proceedings of the 2nd International Workshop on Evidential Assessment of Software Technologies, EAST ’12, Lund, Sweden.10.1145/2372233.2372235 ↗
11. Program Generation and Code Completion Techniques Based on Deep Learning: Literature Review
12. SUS: A quick and dirty usability scale
13. Design and Evaluation of a Short Version of the User Experience Questionnaire (UEQ-S)
14. The One Number You Need to Grow
15. IBM (2023, December 02). IBM SPSS Statistics. Available online: https://www.ibm.com/products/spss-statistics.
16. (2023, December 02). UEQ-Online. Available online: https://www.ueq-online.org/.
17. Lenhard, W., and Lenhard, A. (2022, October 24). Computation of Effect Sizes. Available online: https://www.psychometrica.de/effect_size.html.
18. Xu, F.F., Vasilescu, B., and Neubig, G. (2021). In-IDE Code Generation from Natural Language: Promise and Challenges. arXiv.10.1145/3487569 ↗
19. Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., and Aftandilian, E. (2022). Productivity Assessment of Neural Code Completion. arXiv.10.1145/3520312.3534864 ↗
20. Vaithilingam, P., Zhang, T., and Glassman, E.L. (May, January 29). Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. Proceedings of the Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA ’22, New Orleans, LA, USA.10.1145/3491101.3519665 ↗
21. Svyatkovskiy, A., Zhao, Y., Fu, S., and Sundaresan, N. (2019). Pythia: AI-assisted Code Completion System. arXiv.10.1145/3292500.3330699 ↗
22. Terada, K., and Watanobe, Y. (2019, January 9–10). Code Completion for Programming Education based on Recurrent Neural Network. Proceedings of the 2019 IEEE 11th International Workshop on Computational Intelligence and Applications (IWCIA), Hiroshima, Japan.10.1109/IWCIA47330.2019.8955090 ↗
23. Generative API usage code recommendation with parameter concretization10.1007/s11432-018-9821-9 ↗
24. Schumacher, M.E.H., Le, K.T., and Andrzejak, A. (2020, January 23–29). Improving Code Recommendations by Combining Neural and Classical Machine Learning Approaches. Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW’20, Seoul, Republic of Korea.10.1145/3387940.3391489 ↗
25. Li, J., Huang, R., Li, W., Yao, K., and Tan, W. (2021). Toward Less Hidden Cost of Code Completion with Acceptance and Ranking Models. arXiv.10.1109/ICSME52107.2021.00024 ↗
26. Kalyon, M.S., and Akgul, Y.S. (2021, January 11–13). A Two Phase Smart Code Editor. Proceedings of the 2021 3rd International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA), Ankara, Turkey.10.1109/HORA52670.2021.9461307 ↗
27. Karampatsis, R.M., Babii, H., Robbes, R., Sutton, C., and Janes, A. (July, January 27). Big code != big vocabulary. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, Seoul, Republic of Korea.
28. Nguyen, P.T., Di Rocco, J., Di Ruscio, D., Ochoa, L., Degueule, T., and Di Penta, M. (2019, January 25–31). FOCUS: A Recommender System for Mining API Function Calls and Usage Patterns. Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), Montreal, QC, Canada.10.1109/ICSE.2019.00109 ↗
29. A Graph Sequence Neural Architecture for Code Completion with Semantic Structure Features10.1002/smr.2414 ↗
30. Nguyen, S.V., Nguyen, T.N., Li, Y., and Wang, S. (2019). Combining Program Analysis and Statistical Language Model for Code Statement Completion. arXiv.10.1109/ASE.2019.00072 ↗
31. Zhong, C., Yang, M., and Sun, J. (2019, January 15–18). JavaScript Code Suggestion Based on Deep Learning. Proceedings of the 2019 3rd International Conference on Innovation in Artificial Intelligence, ICIAI 2019, Suzhou, China.10.1145/3319921.3319922 ↗
32. Ciniselli, M., Cooper, N., Pascarella, L., Poshyvanyk, D., Di Penta, M., and Bavota, G. (2021). An Empirical Study on the Usage of BERT Models for Code Completion. arXiv.10.1109/MSR52588.2021.00024 ↗
33. Wang, Y., and Li, H. (2021). Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs. arXiv.10.1609/aaai.v35i16.17650 ↗
34. Arkesteijn, Y., Saldanha, N., and Kostense, B. (2020). Code Completion using Neural Attention and Byte Pair Encoding. arXiv.
35. Hu, X., Men, R., Li, G., and Jin, Z. (2019, January 15–19). Deep-AutoCoder: Learning to Complete Code Precisely with Induced Code Tokens. Proceedings of the 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), Milwaukee, WI, USA.10.1109/COMPSAC.2019.00030 ↗
36. Sobania, D., Briesch, M., and Rothlauf, F. (2022, January 9–13). Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of Github Copilot and Genetic Programming. Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’22, Boston, MA, USA.10.1145/3512290.3528700 ↗
37. Svyatkovskiy, A., Deng, S.K., Fu, S., and Sundaresan, N. (2020). IntelliCode Compose: Code Generation Using Transformer. arXiv.10.1145/3368089.3417058 ↗
38. Schuster, R., Song, C., Tromer, E., and Shmatikov, V. (2020). You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion. arXiv.
39. Kite (2022, October 24). Kite-Free AI Coding Assitant and Code Auto-Complete Plugin. Available online: https://www.kite.com/.
40. Franks, C., Tu, Z., Devanbu, P., and Hellendoorn, V. (2015, January 16–24). CACHECA: A Cache Language Model Based Code Suggestion Tool. Proceedings of the 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Florence, Italy.10.1109/ICSE.2015.228 ↗
41. (2022, October 24). Galois Autocompleter. Available online: https://github.com/MicrosoftDocs/intellicode/.
42. OpenAI (2022, October 24). Better Language Models and Their Implications. Available online: https://openai.com/blog/better-language-models/.
43. OpenAI (2023). GPT-4 Technical Report. arXiv.
44. (2023, November 24). codeium. Available online: https://codeium.com/.
45. Ricca, F., Marchetto, A., and Stocco, A. (2021, January 12–16). AI-based Test Automation: A Grey Literature Analysis. Proceedings of the 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Valencia, Spain.10.1109/ICSTW52544.2021.00051 ↗
46. Grounded Copilot: How Programmers Interact with Code-Generating Models10.1145/3586030 ↗
47. Downey, J. (2008). Career Paths for Programmers: Skills in Senior Software Roles, Cambridge Scholars Publisher.