journal article·2022-01-05Open access

Leveraging AI and Machine Learning for National Student Survey: Actionable Insights from Textual Feedback to Enhance Quality of Teaching and Learning in UK’s Higher Education

Raheel Nawaz, Quanbin Sun, Matthew Shardlow, Georgios Kontonatsios, Naif R. Aljohani, Anna Visvizi, Saeed-Ul Hassan

Applied Sciences· Volume 12· Issue 1

Abstract

Students’ evaluation of teaching, for instance, through feedback surveys, constitutes an integral mechanism for quality assurance and enhancement of teaching and learning in higher education. These surveys usually comprise both the Likert scale and free-text responses. Since the discrete Likert scale responses are easy to analyze, they feature more prominently in survey analyses. However, the free-text responses often contain richer, detailed, and nuanced information with actionable insights. Mining these insights is more challenging, as it requires a higher degree of processing by human experts, making the process time-consuming and resource intensive. Consequently, the free-text analyses are often restricted in scale, scope, and impact. To address these issues, we propose a novel automated analysis framework for extracting actionable information from free-text responses to open-ended questions in student feedback questionnaires. By leveraging state-of-the-art supervised machine learning techniques and unsupervised clustering methods, we implemented our framework as a case study to analyze a large-scale dataset of 4400 open-ended responses to the National Student Survey (NSS) at a UK university. These analyses then led to the identification, design, implementation, and evaluation of a series of teaching and learning interventions over a two-year period. The highly encouraging results demonstrate our approach’s validity and broad (national and international) application potential—covering tertiary education, commercial training, and apprenticeship programs, etc., where textual feedback is collected to enhance the quality of teaching and learning.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Computational and Text Analysis Methods · Evaluation of Teaching Practices

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Operations, Technology, Events and Hospitality Management, Manchester Metropolitan University, Manchester M15 6BH, UK

Department of Computing & Data Science, Birmingham City University, Birmingham B4 7XG, UK

Department of Computing and Mathematics, Manchester Metropolitan University, Manchester M15 6BH, UK

Department of Computer Science, Edge Hill University, Ormskirk L39 4QP, UK

Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia

Effat College of Business, Effat University, Jeddah 21551, Saudi Arabia

Institute of International Studies (ISM), SGH Warsaw School of Economics, Al. Niepodległości 162, 02-554 Warsaw, Poland

References (62)

Show source-provided references
1. The accuracy of student’s evaluations. Study of their evaluative competences
2. Fuentes, A.R., and Ortega, J.G. (2021). An Analytical Perspective of the Reliability of Multi-Evaluation in Higher Education. Mathematics, 9.10.3390/math9111223 ↗
3. Botaccio, L.A., Ortega, J.L.G., Rincón, A.N., and Fuentes, A.R. (2020). Evaluation for Teachers and Students in Higher Education. Sustainability, 12.10.3390/su12104078 ↗
4. Higher Education Funding Council for England (2021, August 20). National Student Survey Results 2016. Available online: http://www.hefce.ac.uk/lt/nss/results/2016/.
5. Evaluating and enhancing quality in higher education teaching practice: A meta review10.1080/03075079.2020.1730315 ↗
6. Paek, S., and Kim, N. (2021). Analysis of Worldwide Research Trends on the Impact of Artificial Intelligence in Education. Sustainability, 13.10.3390/su13147941 ↗
7. The changing nature of teaching and unit evaluations in Australian universities10.1108/09684881211240321 ↗
8. NSS (2021, August 20). National Student Survey. Available online: http://www.hefce.ac.uk/lt/nss/results/2015/.
9. Quality in action: Closing the loop10.1108/09684881011035358 ↗
10. Symons, R. (2006). Listening to the Student Voice at the University of Sydney: Closing the Loop in the Quality Enhancement and Improvement Cycle, Australian Association for Institutional Research Forum. Available online: http://www.aair.org.au/app/webroot/media/pdf/AAIR%20Fora/Forum2006/Symons.pdf.
11. Boring, A., and Ottoboni, K. (2016). Student Evaluations of Teaching (Mostly) Do Not Measure Teaching Effectiveness. Sci. Res., 1–11.10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1 ↗
12. Does the student evaluation of teaching instrument really measure instructors’ teaching effectiveness? An econometric analysis of students’ perceptions in economics courses10.1016/j.eap.2014.05.009 ↗
13. Moving beyond face value: Re-envisioning higher education evaluation as a generator of professional knowledge10.1080/02602938.2011.565114 ↗
14. Student Evaluations of Teaching: Are They Related to What Students Learn? A Meta-Analysis and Review of the Literature10.1177/0273475308324086 ↗
15. Are Student Evaluations of Teaching Effectiveness Valid for Measuring Student Learning Outcomes in Business Related Classes? A Neural Network and Bayesian Analyses10.1007/s11162-011-9229-0 ↗
16. A comparison of open-ended and closed questions in the prediction of mental health10.1007/s11135-011-9597-8 ↗
17. Ulmeanu, M.-E., Doicin, C.-V., and Spânu, P. (2021). Comparative Evaluation of Sustainable Framework in STEM Intensive Programs for Secondary and Tertiary Education. Sustainability, 13.10.3390/su13020978 ↗
18. Closed and Open-Ended Question Tools in a Telephone Survey About “The Good Teacher”: An Example of a Mixed Method Study10.1177/1558689808331036 ↗
19. The meaning of prompt feedback and other student perceptions of feedback: Should National Student Survey scores be taken at face value?10.11120/ened.2011.06010031 ↗
20. The National Student Survey: Development, findings and implications10.1080/03075070701573757 ↗
21. Exploring the ‘wicked’ problem of student dissatisfaction with assessment and feedback in higher education10.1080/23752696.2019.1644659 ↗
22. Using National Student Survey (NSS) qualitative data and social identity theory to explore students’ experiences of assessment and feedback10.1080/23752696.2019.1601500 ↗
23. Coherence between text comments and the quantitative ratings in the UK’s National Student Survey10.1080/0309877X.2014.1000281 ↗
24. Student evaluations of teaching: Improving teaching quality in higher education
25. National Student Survey: Are differences between universities and courses reliable and meaningful?10.1080/03054985.2010.491179 ↗
26. Measuring student satisfaction at a UK university10.1108/09684880610678568 ↗
27. From performativity to professionalism: Lecturers’ responses to student feedback10.1080/13562510903050228 ↗
28. Honesty on student evaluations of teaching: Effectiveness, purpose, and timing matter!
29. On the Validity of Student Evaluation of Teaching: The State of the Art10.3102/0034654313496870 ↗
30. Oberrauch, A., Mayr, H., Nikitin, I., Bügler, T., Kosler, T., and Vollmer, C. (2021). I Wanted a Profession That Makes a Difference—An Online Survey of First-Year Students’ Study Choice Motives and Sustainability-Related Attributes. Sustainability, 13.10.3390/su13158273 ↗
31. A revalidation of the SET37 questionnaire for student evaluations of teaching10.1080/03055690902880299 ↗
32. A Multilevel Factor Analysis of Students’ Evaluations of Teaching10.1177/0013164404268667 ↗
33. Whose feedback? A multilevel analysis of student completion of end-of-term teaching evaluations10.1080/02602938.2015.1044421 ↗
34. An integrated strategy for the analysis of student evaluation of teaching: From descriptive measures to explanatory models10.1007/s11135-016-0432-0 ↗
35. A comprehensive empirical comparison of modern supervised classification and feature selection methods for text categorization10.1002/asi.23110 ↗
36. Probabilistic Latent Semantic Indexing10.1145/3130348.3130370 ↗
37. Wallace, B.C., Trikalinos, T.A., Lau, J., Brodley, C., and Schmid, C.H. (2010). Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinform., 11.10.1186/1471-2105-11-55 ↗
38. Exploratory Visual Analysis and Interactive Pattern Extraction from Semi-Structured Data10.1145/2812115 ↗
39. Mining concepts of health responsibility using text mining and exploratory graph analysis
40. Semantic Web Search Results Clustering Using Lingo and WordNet
41. X-ray, lensing and Sunyaev-Zel’dovich triaxial analysis of Abell 1835 out to R200
42. Text analysis of open-ended survey responses: A complementary method to preference mapping10.1016/S0950-3293(02)00011-3 ↗
43. Analyses of open-ended questions by renormalized associativities and textual networks: A study of perception of minerality in wine10.1016/j.foodqual.2015.06.013 ↗
44. Structural Topic Models for Open-Ended Survey Responses10.1111/ajps.12103 ↗
45. Krosnick, J.A. (2018). Questionnaire design. The Palgrave Handbook of Survey Research, Springer International Publishing.10.1007/978-3-319-54395-6_53 ↗
46. Open-ended vs. close-ended questions in web questionnaires
47. Measuring Emotion in Open-ended Survey Responses: An Application of Textual Data Analysis
48. Determinants of First-Year Student Identity and satisfaction in higher education: A quantitative case study10.1177/2158244019846689 ↗
49. Perceived Social Support and Well Being: First-Year Student Experience in University10.5539/ies.v7n13p261 ↗
50. An Analysis of the Effects of Self-Esteem, Social Support, and Participation in Student Support Services on Students’ Adjustment and Commitment to College10.2190/C0T7-YX50-F71V-00CW ↗
51. It was nothing to do with the university, it was just the people: The role of social support in the first-year experience of higher education10.1080/03075070500340036 ↗
52. Support-Vector Networks10.1007/BF00994018 ↗
53. Wang, S., and Manning, C.D. (2012, January 8). Baselines and bigrams: Simple, good sentiment and topic classification. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Korea.
54. Ensemble of keyword extraction methods and classifiers in text classification10.1016/j.eswa.2016.03.045 ↗
55. Latent dirichlet allocation
56. Lau, J.H., Newman, D., and Baldwin, T. (2014, January 26–30). Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, Gothenburg, Sweden.10.3115/v1/E14-1056 ↗
57. Closing the feedback loop: Ensuring effective action from student feedback10.1080/13583883.2003.9967099 ↗
58. Hettrick, S. (2013). What’s wrong with computer scientists?. Softw. Sustain. Inst., Available online: https://www.software.ac.uk/blog/2016-10-06-whats-wrong-computer-scientists.
59. (2021, December 22). Why Your Computer Science Degree Won’t Get You a Job. Available online: https://targetjobs.co.uk/career-sectors/it-and-technology/advice/323039-why-your-computer-science-degree-wont-get-you-an-it-job.
60. A performance indicator of teaching quality in higher education: The Course Experience Questionnaire10.1080/03075079112331382944 ↗
61. Silveira, N., Dozat, T., De Marneffe, M.C., Bowman, S.R., Connor, M., Bauer, J., and Manning, C.D. (2014, January 26–31). A gold standard dependency corpus for English. Proceedings of the LERC, Reykjavik, Iceland.
62. Wissler, L., Almashraee, M., Monett, D., and Paschke, A. (2014, January 26–27). The gold standard in corpus annotation. Proceedings of the IEEE GSC, Passau, Germany.