Towards a Code-Based Assessment of Computational Thinking Skills
Source abstract
Abstract
Computational thinking (CT) skills are central to higher-education programming, yet assessing them at scale is challenging. Automated source-code analysis is a promising path, but its validity at university level is still poorly understood. We examined how far code-based metrics can capture traces of two CT skills: abstraction and algorithmic thinking. In a controlled study, 44 students enrolled in a Data Structures course completed four programming exercises designed to elicit evidence of the two target computational thinking skills. Each submission was assessed using three independent sources: an automated score based on structural code metrics, a rubric-based score by experts, and a CT self-efficacy scale. The three sources were compared using Spearman correlations, quartile analyses, and qualitative inspection of fourteen concordant and divergent cases. At the per-student level, the three sources did not converge for either skill. Per-exercise, one converged with expert evaluation, while another showed that the sources captured distinct, complementary facets of the skill—an exercise-design effect. Qualitative cases reinforced this view, illustrating what each source reads: from structural shape to semantic correctness. Automated code-based assessment of CT can provide informative structural evidence, but works best as one element of a multi-source approach in which structural signals complement, rather than replace, expert evaluation.
Source: crossref · Quality: baseline
Research context
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Technology Management, Costa Rica Institute of Technology, Campus Cartago, Cartago 30101, Costa Rica
School of Mathematics, Costa Rica Institute of Technology, Campus Cartago, Cartago 30101, Costa Rica
School of Computing, Costa Rica Institute of Technology, Campus Cartago, Cartago 30101, Costa Rica
School of Software Engineering, Universidad CENFOTEC, San José 10104, Costa Rica
Department of the Doctoral Program in Engineering, Costa Rica Institute of Technology, Campus Cartago, Cartago 30101, Costa Rica
References (85)
Show source-provided references
1. Computational thinking10.1145/1118178.1118215 ↗
2. Demystifying computational thinking10.1016/j.edurev.2017.09.003 ↗
3. Remaining Trouble Spots with Computational Thinking10.1145/2998438 ↗
4. Assessing computational thinking: A systematic review of empirical studies10.1016/j.compedu.2019.103798 ↗
5. A scoping review of empirical research on recent computational thinking assessments10.1007/s10956-019-09799-3 ↗
6. De Jong, I., and Jeuring, J. (2020, January 19–22). Computational thinking interventions in higher education: A scoping literature review of interventions used to teach computational thinking. Proceedings of the 20th Koli Calling International Conference on Computing Education Research, Koli, Finland.10.1145/3428029.3428055 ↗
7. De Jong, I., Sichterman, B., and Jeuring, J. (2024). Use of Computational Thinking Skills when solving Bebras Tasks: A Think-aloud Study. Proceedings of the 24th Koli Calling International Conference on Computing Education Research, Koli Calling ’24, Association for Computing Machinery.10.1145/3699538.3699543 ↗
8. Measuring the difficulty of code comprehension tasks using software metrics
9. How difficult are novice code writing tasks? A software metrics approach
10. Dr. Scratch: Automatic Analysis of Scratch Projects to Assess and Foster Computational Thinking
11. A Scoping Review of Computational Thinking Assessments in Higher Education10.1007/s12528-021-09305-y ↗
12. Coding and Computational Thinking Across the Curriculum: A Review of Educational Outcomes10.3102/00346543241241327 ↗
13. Computational thinking in K-12: A review of the state of the field10.3102/0013189X12463051 ↗
14. Fundamentos y perspectivas del pensamiento computacional: Un análisis integral para la investigación futura
15. Review on teaching and learning of computational thinking through programming: What is next for K-12?10.1016/j.chb.2014.09.012 ↗
16. Research Notebook: Computational Thinking—What and Why
17. Selby, C., and Woollard, J. (2013). Computational Thinking: The Developing Definition, University of Southampton.
18. Brennan, K., and Resnick, M. (2012, January 13–17). New frameworks for studying and assessing the development of computational thinking. Proceedings of the 2012 Annual Meeting of the American Educational Research Association, Vancouver, BC, Canada.
19. DWES: A Dynamic Weighted Evaluation System for Scratch based on Computational Thinking
20. Computational Thinking and User Interfaces: A Systematic Review10.1109/TE.2022.3159765 ↗
21. Zhang, W., Zeng, X., Ming, D., and Wang, J. (2022, January 9–11). Research on the Construction of Evaluation Indicators of Students’ Computational Thinking Based on Spectral Clustering. Proceedings of the 2022 10th International Conference on Information and Education Technology (ICIET), Matsue, Japan.10.1109/ICIET55102.2022.9779003 ↗
22. Assessing computational thinking: An overview of the field10.1080/08993408.2021.1918380 ↗
23. Bandura, A. (1997). Self-Efficacy: The Exercise of Control, Freeman.
24. De Jong, I., and Jeuring, J. (2022). Developing a Self-efficacy Scale for Computational Thinking (CT-SES). Proceedings of the 22nd Koli Calling International Conference on Computing Education Research, Association for Computing Machinery.10.1145/3564721.3565954 ↗
25. Maloney, J.H., Peppler, K., Kafai, Y., Resnick, M., and Rusk, N. (2008). Programming by choice: Urban youth learning programming with scratch. Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’08, Association for Computing Machinery.10.1145/1352135.1352260 ↗
26. Yang, S., Domeniconi, C., Revelle, M., Sweeney, M., Gelman, B.U., Beckley, C., and Johri, A. (2015). Uncovering Trajectories of Informal Learning in Large Online Communities of Creators. Proceedings of the Second ACM Conference on Learning @ Scale, L@S ’15, Association for Computing Machinery.10.1145/2724660.2724674 ↗
27. Robles, G., Moreno-León, J., Aivaloglou, E., and Hermans, F. (2017, January 21–21). Software clones in Scratch projects: On the presence of copy-and-paste in computational thinking learning. Proceedings of the 2017 IEEE 11th International Workshop on Software Clones (IWSC), Klagenfurt, Austria.10.1109/IWSC.2017.7880506 ↗
28. Assessing computational thinking process using a multiple evaluation approach10.1016/j.ijcci.2018.10.004 ↗
29. The Use of Cognitive Diagnostic Modeling in the Assessment of Computational Thinking10.1177/23328584221081256 ↗
30. The competent computational thinking test: Development and validation of an unplugged computational thinking test for upper primary school10.1177/07356331221081753 ↗
31. Hijón-Neira, R., Pizarro, C., French, J., Palacios-Alonso, D., and Çoban, E. (2024). Computational Thinking Measurement of CS University Students. Appl. Sci., 14.10.20944/preprints202405.1203.v1 ↗
32. Kong, S.C., and Abelson, H. (2019). Combining Assessment Tools for a Comprehensive Evaluation of Computational Thinking Interventions. Computational Thinking Education, Springer.10.1007/978-981-13-6528-7 ↗
33. Aivaloglou, E., and Hermans, F. (2016). How Kids Code and How We Know: An Exploratory Study on the Scratch Repository. Proceedings of the 2016 ACM Conference on International Computing Education Research, ICER ’16, Association for Computing Machinery.10.1145/2960310.2960325 ↗
34. Impact of Programming Exposure on the Development of Computational Thinking Capabilities: An Empirical Study10.1109/ACCESS.2020.2987254 ↗
35. Benedetti, E., Alpizar-Chacon, I., and Jeuring, J. (2026). The Use of Computational Thinking Skills, Difficulties, and Strategies of Introductory Programming Students Solving Bebras Tasks. arXiv.
36. Aho, A.V., Lam, M.S., Sethi, R., and Ullman, J.D. (2006). Compilers: Principles, Techniques, and Tools, Addison Wesley. [2nd ed.].
37. Tool for generating code metrics for C# source code using abstract syntax tree technique10.1145/2507288.2507312 ↗
38. Pietrzak, B., and Walter, B. (2006). Leveraging code smell detection with inter-smell relations. Proceedings of the 7th International Conference on Extreme Programming and Agile Processes in Software Engineering, XP’06, Springer.10.1007/11774129_8 ↗
39. Human–computer interaction in evolutionary visual software analytics10.1016/j.chb.2012.01.013 ↗
40. Knowledge discovery in software teams by means of evolutionary visual software analytics10.1016/j.scico.2015.09.005 ↗
41. Agrahari, V., and Chimalakonda, S. (2020, January 6–9). AST[AR]—Towards Using Augmented Reality and Abstract Syntax Trees for Teaching Data Structures To Novice Programmers. Proceedings of the 2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT), Tartu, Estonia.10.1109/ICALT49669.2020.00100 ↗
42. Ng, K., Warren, M., Golde, P., and Hejlsberg, A. (2011). The Roslyn Project, Exposing the C# and VB Compiler’s Code Analysis, Microsoft.
43. Leiton-Jimenez, J., and Barboza-Artavia, L.A. (2021). Método Para Convertir Código Fuente Escrito en Diversos Lenguajes de Programación a un Lenguaje Universal. [Master’s Thesis, Costa Rica Institute of Technology]. Available online: https://repositoriotec.tec.ac.cr/bitstream/handle/2238/13420/TF8984_BIB303385_Jason_Leiton_y_Luis_Barboza.pdf.
44. Skalka, J., and Drlik, M. (2020). Automated Assessment and Microlearning Units as Predictors of At-Risk Students and Students’ Outcomes in the Introductory Programming Courses. Appl. Sci., 10.10.3390/app10134566 ↗
45. Jevtić, M., Mladenović, S., and Granić, A. (2023). Source Code Analysis in Programming Education: Evaluating Learning Content with Self-Organizing Maps. Appl. Sci., 13.10.3390/app13095719 ↗
46. Gonzalez-Torres, A., Sancho-Chavarria, L., Zuniga-Cespedes, M., Monge-Fallas, J., and Navas-Su, J. (2021, January 15–17). A strategy to assess computational thinking. Proceedings of the 2021 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA.10.1109/CSCI54926.2021.00222 ↗
47. Pacheco-Portuguez, S., Gonzalez-Torres, A., Sancho-Chavarria, L., Trejos-Zelaya, I., Monge-Fallas, J., Navas-Su, J., Cañas, A.J., Rodríguez, A., and Chinchilla Angulo, C. (2022, January 1–4). A method for assessing computational thinking in students using source code analysis. Proceedings of the 2022 International Conference on Advanced Learning Technologies (ICALT), Bucharest, Romania.10.1109/ICALT55010.2022.00050 ↗
48. Gonzalez-Torres, A., Ramirez-Trejos, E., Sancho-Chavarria, L., Navas-Su, J., Garita, C., and Monge-Fallas, J. (2023, January 24–27). Towards the Assessment of Basic Computational Thinking Skills Using Syntactic Analysis Techniques. Proceedings of the 2023 Congress in Computer Science, Computer Engineering, & Applied Computing (CSCE), Las Vegas, NV, USA.10.1109/CSCE60160.2023.00181 ↗
49. Ramírez-Trejos, E., González-Torres, A., Sancho-Chavarría, L., Navas-Sú, J., Garita, C., and Monge-Fallas, J. (2023, January 23–26). Automatic Assessment of Programming Exercises Using Syntactic Analysis. Proceedings of the 2023 42nd IEEE International Conference of the Chilean Computer Science Society (SCCC), Concepcion, Chile.10.1109/SCCC59417.2023.10315732 ↗
50. Pacheco-Portuguez, S., Gonzalez-Torres, A., Sancho-Chavarria, L., Trejos-Zelaya, I., and Monge-Fallas, J. (2025, January 23–26). Pattern recognition in programming to support the evaluation of Computational Thinking. Proceedings of the 2025 IEEE Engineering Education World Conference (EDUNINE), Montevideo, Uruguay.10.1109/EDUNINE62377.2025.10981362 ↗
51. Predicting the understandability of computational notebooks through code metrics analysis10.1007/s10664-025-10651-z ↗
52. A complexity measure10.1109/TSE.1976.233837 ↗
53. Campbell, G.A. (2018). Cognitive complexity: An overview and evaluation. Proceedings of the 2018 International Conference on Technical Debt, TechDebt ’18, Association for Computing Machinery.10.1145/3194164.3194186 ↗
54. Muñoz Barón, M., Wyrich, M., and Wagner, S. (2020). An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability. Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), ESEM ’20, Association for Computing Machinery.10.1145/3382494.3410636 ↗
55. On the relationship between source-code metrics and cognitive load: A systematic tertiary review10.1016/j.jss.2023.111619 ↗
56. Hao, G., Hijazi, H., Durães, J., Medeiros, J., Couceiro, R., Lam, C.T., Teixeira, C., Castelhano, J., Castelo Branco, M., and Carvalho, P. (2023). On the accuracy of code complexity metrics: A neuroscience-based guideline for improvement. Front. Neurosci., 16.10.3389/fnins.2022.1065366 ↗
57. 40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study
58. Cognitive Load During Problem Solving: Effects on Learning10.1207/s15516709cog1202_4 ↗
59. Duran, R., Sorva, J., and Leite, S. (2018). Towards an Analysis of Program Complexity From a Cognitive Perspective. Proceedings of the 2018 ACM Conference on International Computing Education Research, ICER ’18, Association for Computing Machinery.10.1145/3230977.3230986 ↗
60. Cognitive Complexity Beyond Generalization: A Subjective Rating for the Human Comprehension10.1051/itmconf/20225001003 ↗
61. Early Career Developers’ Perceptions of Code Understandability: A Study of Complexity Metrics10.1109/ACCESS.2025.3585777 ↗
62. Loksa, D., Ko, A.J., Jernigan, W., Oleson, A., Mendez, C.J., and Burnett, M.M. (2016). Programming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, Association for Computing Machinery.10.1145/2858036.2858252 ↗
63. Loksa, D., and Ko, A.J. (2016). The Role of Self-Regulation in Programming Problem Solving Process and Success. Proceedings of the 2016 ACM Conference on International Computing Education Research, ICER ’16, Association for Computing Machinery.10.1145/2960310.2960334 ↗
64. Notional machines and introductory programming education10.1145/2483710.2483713 ↗
65. Biggs, J.B., and Collis, K.F. (1982). Evaluating the Quality of Learning: The SOLO Taxonomy (Structure of the Observed Learning Outcome), Academic Press.
66. Enhancing Teaching Through Constructive Alignment10.1007/BF00138871 ↗
67. Lister, R., Simon, B., Thompson, E., Whalley, J.L., and Prasad, C. (2006). Not Seeing the Forest for the Trees: Novice Programmers and the SOLO Taxonomy. Proceedings of the 11th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ITICSE ’06, Association for Computing Machinery.10.1145/1140124.1140157 ↗
68. Ginat, D., and Menashe, E. (2015). SOLO Taxonomy for Assessing Novices’ Algorithmic Design. Proceedings of the 46th ACM Technical Symposium on Computer Science Education, SIGCSE ’15, Association for Computing Machinery.10.1145/2676723.2677311 ↗
69. Izu, C., Weerasinghe, A., and Pope, C. (2016). A Study of Code Design Skills in Novice Programmers using the SOLO taxonomy. Proceedings of the 2016 ACM Conference on International Computing Education Research, ICER ’16, Association for Computing Machinery.10.1145/2960310.2960324 ↗
70. Castro, F.E.V., and Fisler, K. (2017). Designing a multi-faceted SOLO taxonomy to track program design skills through an entire course. Proceedings of the 17th Koli Calling International Conference on Computing Education Research, Koli Calling ’17, Association for Computing Machinery.10.1145/3141880.3141891 ↗
71. Decker, A., Margulieux, L.E., and Morrison, B.B. (2019). Using the SOLO Taxonomy to Understand Subgoal Labels Effect in CS1. Proceedings of the 2019 ACM Conference on International Computing Education Research, ICER ’19, Association for Computing Machinery.10.1145/3291279.3339405 ↗
72. Sancho-Chavarria, L., Alpizar-Chacon, I., Esquivel-Vargas, H., Monge-Fallas, J., Gonzalez-Torres, A., and Trejos-Zelaya, I. (2025, January 26–28). A Common Framework for Automatically Assessing Computational Thinking Skills. Proceedings of the 2025 IEEE 43rd Central America and Panama Convention (CONCAPAN XLIII), San Salvador, El Salvador.10.1109/CONCAPAN66820.2025.11512501 ↗
73. Cohen, L., Manion, L., and Morrison, K. (2017). Research Methods in Education, Routledge.10.4324/9781315456539 ↗
74. Software reuse: Metrics and models10.1145/234528.234531 ↗
75. The Distribution of the Flora in the Alpine Zone10.1111/j.1469-8137.1912.tb05611.x ↗
76. Felleisen, M., Findler, R.B., Flatt, M., and Krishnamurthi, S. (2018). How to Design Programs: An Introduction to Programming and Computing, MIT Press. [2nd ed.].
77. Exploring computer science students’ continuance intentions to use Kattis10.1007/s10639-017-9658-2 ↗
78. Wing, J.M. (2008). Computational thinking and thinking about computing. Proceedings of the 22nd IEEE International Symposium on Parallel and Distributed Processing, IPDPS 2008, Miami, FL, USA, 14–18 April 2008, IEEE.10.1109/IPDPS.2008.4536091 ↗
79. Developing a Two-Dimensional Categorization System for Educational Tasks in Informatics10.15388/Informatica.2017.119 ↗
80. Selby, C.C. (2015). Relationships: Computational thinking, pedagogy of programming, and Bloom’s Taxonomy. Proceedings of the Workshop in Primary and Secondary Computing Education, WiPSCE ’15, Association for Computing Machinery.10.1145/2818314.2818315 ↗
81. The Measurement of Observer Agreement for Categorical Data10.2307/2529310 ↗
82. Fenton, N., and Bieman, J. (2014). Software Metrics: A Rigorous and Practical Approach, CRC Press.10.1201/b17461 ↗
83. Stegeman, M., Barendsen, E., and Smetsers, S. (2014). Towards an empirically validated model for assessment of code quality. Proceedings of the 14th Koli Calling International Conference on Computing Education Research, Koli Calling ’14, Association for Computing Machinery.10.1145/2674683.2674702 ↗
84. Visual programming languages integrated across the curriculum in elementary school: A two year case study using “Scratch” in five schools10.1016/j.compedu.2016.03.003 ↗
85. Insights into computational thinking from think-aloud interviews: A systematic review10.1002/acp.4029 ↗