Personalized AI-Directed Tutoring for Oral Proficiency Enhancement in Language Education
Source abstract
Abstract
Generative AI offers transformative potential for scalable, personalized, and dynamic language education, particularly in enhancing oral proficiency among young learners. However, effective deployment remains challenging due to limited resources for some languages, the need for age-appropriate content and tools, and the importance of respecting cultural relevance. In this paper, we introduce LEARN (Language Evaluation via question Answer generation from caRtooNs), a culturally grounded multilingual visual dialogue system designed to support oral proficiency in three of Singapore’s official languages: Mandarin, Bahasa Melayu, and Tamil. English, as the lingua franca, is excluded. LEARN integrates a teacher-facing module for curriculum-aligned visual question-answering task creation and a student-facing module for voice-driven adaptive dialogue, optimized for children’s speech. Unlike existing platforms, LEARN prioritizes cultural relevance and low-resource language support, helping address gaps in heritage language preservation. Pilot studies with students demonstrate significant improvements in engagement and vocabulary acquisition. Designed for classroom as well as home use, LEARN presents a scalable AI-driven language tutoring framework.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Speech and dialogue systems · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
ICT Cluster, Singapore Institute of Technology, Singapore 828608, Singapore
College of Computing and Data Science, Nanyang Technological University, Singapore 639798, Singapore
National Engineering Research Center of Speech and Language Information Processing, The University of Science and Technology of China, Hefei 230052, China
Funding
National Research Foundation, Singapore
References (58)
Show source-provided references
1. Promises and challenges of generative artificial intelligence for human learning10.1038/s41562-024-02004-5 ↗
2. Chakraborty, S. (2024). Generative AI in Modern Education Society. arXiv.
3. Exploring audience engagement with ChatGPT-related content on YouTube: Implications for content creators and AI tool developers10.1016/j.digbus.2023.100071 ↗
4. ID. 8: Co-Creating visual stories with Generative AI10.1145/3672277 ↗
5. Alenezi, M., and Akour, M. (2025). AI-Driven Innovations in Software Engineering: A Review of Current Practices and Future Directions. Appl. Sci., 15.10.3390/app15031344 ↗
6. Al Naqbi, H., Bahroun, Z., and Ahmed, V. (2024). Enhancing work productivity through generative artificial intelligence: A comprehensive literature review. Sustainability, 16.10.3390/su16031166 ↗
7. Jurenka, I., Kunesch, M., McKee, K.R., Gillick, D., Zhu, S., Wiltberger, S., Phal, S.M., Hermann, K., Kasenberg, D., and Bhoopchand, A. (2024). Towards responsible development of generative AI for education: An evaluation-driven approach. arXiv.
8. Azevedo, J.P.W.D., Rogers, F.H., Ahlgren, S.E., Cloutier, M.H., Chakroun, B., Chang, G.C., Mizunoya, S., Reuge, N.J., Brossard, M., and Bergmann, J.L. (2021). The State of the Global Education Crisis: A Path to Recovery (Vol. 2): Executive Summary (English), World Bank Group. Available online: http://documents.worldbank.org/curated/en/184161638768635066.
9. Xia, Y., Shin, S.Y., and Kim, J.C. (2024). Cross-cultural intelligent language learning system (CILS): Leveraging AI to facilitate language learning strategies in cross-cultural communication. Appl. Sci., 14.10.20944/preprints202405.0425.v1 ↗
10. Living in harmony: The negotiation of intergenerational family language policy in Singapore10.1016/j.langcom.2021.11.004 ↗
11. Dennison, D.V., Ahtisham, B., Chourasia, K., Arora, N., Singh, R., Kizilcec, R.F., Nambi, A., Ganu, T., and Vashistha, A. (2025). Teacher-AI Collaboration for Curating and Customizing Lesson Plans in Low-Resource Schools. arXiv.
12. Jiao, J., Afroogh, S., Chen, K., Murali, A., Atkinson, D., and Dhurandhar, A. (2025). LLMs and Childhood Safety: Identifying Risks and Proposing a Protection Framework for Safe Child-LLM Interaction. arXiv.
13. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions
14. Survey of cultural awareness in language models: Text and beyond10.1162/COLI.a.14 ↗
15. Qian, M., McLoughlin, I., Quo, W., and Dai, L. (2016). Mismatched training data enhancement for automatic recognition of children’s speech using DNN-HMM. Proceedings of the 2016 10th International Symposium on Chinese Spoken Language Processing (ISCSLP), IEEE.10.1109/ISCSLP.2016.7918386 ↗
16. Limitations in speech recognition for young adults with down syndrome10.1007/s10209-025-01197-4 ↗
17. Zhang, B., Latiff, N.A.A., Kan, J., Tong, R., Soh, D., Miao, X., and McLoughlin, I. (2025, January 17–21). Automated evaluation of children’s speech fluency for low-resource languages. Proceedings of the Interspeech, Rotterdam, The Netherlands.10.21437/Interspeech.2025-1358 ↗
18. Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., and Parikh, D. (2016, January 27–30). Yin and Yang: Balancing and Answering Binary Visual Questions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.542 ↗
19. Artificial intelligence in science education (2013–2023): Research trends in ten years10.1007/s10956-023-10077-6 ↗
20. Dascalescu, S., Dumitran, A.M., and Vasiluta, M.A. (2025). Leveraging Generative AI for Enhancing Automated Assessment in Programming Education Contests. arXiv.10.18653/v1/2025.bea-1.7 ↗
21. Programming education and learner motivation in the age of generative AI: Student and educator perspectives
22. Artificial intelligence for mental health and mental illnesses: An overview10.1007/s11920-019-1094-0 ↗
23. Impact of GPT-Driven Teaching Assistants in VR Learning Environments10.1109/TLT.2025.3539179 ↗
24. Denny, P., MacNeil, S., Savelka, J., Porter, L., and Luxton-Reilly, A. (2024). Desirable characteristics for ai teaching assistants in programming education. Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, Association for Computing Machinery.10.1145/3649217.3653574 ↗
25. Kotsis, K.T. (2024). ChatGPT as teacher assistant for physics teaching. EIKI J. Eff. Teach. Methods, 2.10.59652/jetm.v2i4.283 ↗
26. Maity, S., and Deroy, A. (2024). Generative ai and its impact on personalized intelligent tutoring systems. arXiv.10.36227/techrxiv.172954174.48041095/v1 ↗
27. Exploring applications of ChatGPT to English language teaching: Opportunities, challenges, and recommendations10.55593/ej.27107int ↗
28. The impact of chatbots based on large language models on second language vocabulary acquisition10.1016/j.heliyon.2024.e25370 ↗
29. Improving automatic vqa evaluation using large language models10.1609/aaai.v38i5.28212 ↗
30. Harnessing AI for Education 4.0: Drivers of Personalized Learning10.34190/ejel.22.5.3467 ↗
31. Personalized education and artificial intelligence in the United States, China, and India: A systematic review using a human-in-the-loop model10.1016/j.caeai.2022.100068 ↗
32. Liu, Z., Lin, G., Tan, H.L., Zhang, H., Lu, Y., Gao, X., Yin, S.X., He, S., Goh, H.H., and Wong, L.H. (2025). SingaKids: A multilingual multimodal dialogic tutor for language learning. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), Association for Computational Linguistics.10.18653/v1/2025.acl-industry.86 ↗
33. Leveraging an LLM-enhanced bilingual conversational agent for EFL children’s dialogic reading: Insights from children, parents, and educators10.1016/j.caeai.2025.100484 ↗
34. Krashen, S.D. (1985). The Input Hypothesis: Issues and Implications, Longman.
35. Vygotsky, L.S. (1978). Mind in Society: The Development of Higher Psychological Processes, Harvard University Press.
36. Constructing a language: A usage-based theory of language acquisition10.1017/S0272263104363059 ↗
37. Long, M. (1996). The role of the linguistic environment in second language acquisition. Handbook of Second Language Acquisition, Academic Press.10.1016/B978-012589042-7/50015-3 ↗
38. Paivio, A. (1990). Mental Representations: A Dual Coding Approach, Oxford University Press.10.1093/acprof:oso/9780195066661.001.0001 ↗
39. The pedagogical values of cartoons
40. Tushar, P., Pandey, E., Austria, L.D.B., Loo, Y.Y., Lim, J.H., Atmosukarto, I., and Lock, D.S.C. (2025). MerCulture: A Comprehensive Benchmark to Evaluate Vision-Language Models on Cultural Understanding in Singapore. Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR) Workshops, The Computer Vision Foundation/IEEE Computer Society.10.1109/CVPRW67362.2025.00061 ↗
41. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., and Tang, J. (2025). Qwen2. 5-vl technical report. arXiv.
42. Girish, D., Singh, V., and Ralescu, A. (2020). Understanding action recognition in still images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, The Computer Vision Foundation/IEEE Computer Society.10.1109/CVPRW50498.2020.00193 ↗
43. Kong, A., Zhao, S., Chen, H., Li, Q., Qin, Y., Sun, R., Zhou, X., Wang, E., and Dong, X. (2023). Better zero-shot reasoning with role-play prompting. arXiv.10.18653/v1/2024.naacl-long.228 ↗
44. Fairness-guided few-shot prompting for large language models
45. Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W., and Zhao, T. (2023). Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv.
46. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
47. Srivastava, K., Zaid, M., Sowjanya, P., and Adavanne, S. (2025, December 01). Krutrim Translate. Available online: https://github.com/ola-krutrim/KrutrimTranslate.
48. Gala, J., Chitale, P.A., Raghavan, A.K., Gumma, V., Doddapaneni, S., Kumar, A., Nawale, J.A., Sujatha, A., Puduppully, R., and Raghavan, V. (2023). IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages. Trans. Mach. Learn. Res.
49. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. Proceedings of the International Conference on Machine Learning, ICML.
50. Zhang, B., Latiff, N.A.A., Tong, R., Soh, D., and McLoughlin, I. (2025). Scsmt: A Multilingual Children’s Speech Corpus for Singapore’s Mother Tongues. Proceedings of the 2025 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), IEEE.10.1109/APSIPAASC65261.2025.11249271 ↗
51. Husein, Z. (2025, October 01). Malaya-Speech. Available online: https://github.com/mesolitica/malaya-speech.
52. Zhang, H., Yuan, T., Chen, J., Li, X., Zheng, R., Huang, Y., Chen, X., Gong, E., Chen, Z., and Hu, X. (2022). PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, Association for Computational Linguistics.10.18653/v1/2022.naacl-demo.12 ↗
53. Kumar, G.K., Praveen, S., Kumar, P., Khapra, M.M., and Nandakumar, K. (2023). Towards building text-to-speech systems for the next billion users. Proceedings of the ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE.10.1109/ICASSP49357.2023.10096069 ↗
54. Desplanques, B., Thienpondt, J., and Demuynck, K. (2020). Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. arXiv.10.21437/Interspeech.2020-2650 ↗
55. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics; ACL ’02, Association for Computational Linguistics.10.3115/1073083.1073135 ↗
56. NVIDIA Corporation (2026, February 23). Triton Inference Server: An Optimized Cloud and Edge Inferencing Solution. Available online: https://github.com/triton-inference-server.
57. Ramírez, S. (2025, October 01). FastAPI. Available online: https://fastapi.tiangolo.com/.
58. Xuereb, E. (2025). Exploring NLP Models for Generative Storytelling. [Master’s Thesis, Utrecht University].