Retrieval-Augmented Vision–Language Agents for Child-Centered Encyclopedia Learning
Source abstract
Abstract
This study introduces an Encyclopedic Agent for children’s learning that integrates multimodal retrieval with retrieval-augmented generation (RAG). To support this framework, we construct a dataset of 9524 Wikipedia pages covering 935 encyclopedia topics, each converted into images with associated topical queries and explanations. Based on this dataset, we fine-tune SigLIP, a vision–language retrieval model, using LoRA adaptation on 8484 training pairs, with 1040 reserved for testing. Experimental results show that the fine-tuned SigLIP significantly outperforms baseline models such as ColPali in both accuracy and latency, enabling efficient and precise document-image retrieval. Combined with GPT-5 for response generation, the Encyclopedic Agent delivers illustrated, interactive Q&A that is more accessible and engaging for children compared to traditional text-only methods. These findings highlight the feasibility of applying multimodal retrieval and RAG to educational agents, offering new possibilities for personalized, child-centered learning in domains such as science, history, and the arts.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Automated Systems · Semantic Web and Ontologies · Speech and dialogue systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Media & Communication, Kangwon National University, Chuncheon 24341, Republic of Korea
School of Information Science and Technology, Hangzhou Normal University, Hangzhou 311121, China
Center for Engineering and Scientific Computation, Zhejiang University, Hangzhou 310058, China
References (47)
Show source-provided references
1. OpenAI (2023). GPT-4 Technical Report. arXiv.
2. Google Gemini Team (2025, September 14). Gemini: A Family of Highly Capable Multimodal Models. Available online: https://storage.googleapis.com/deepmindmedia/gemini/gemini_1_report.pdf.
3. For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning10.1037/0022-0663.86.3.389 ↗
4. Multimodality of AI for Education: Toward Artificial General Intelligence10.1109/TLT.2025.3574466 ↗
5. Ji, H., Qiu, S., Xin, S., Han, S., Chen, Z., Zhang, D., Wang, H., and Yao, H. (2025). From EduVisBench to EduVisAgent: A Benchmark and Multi-Agent Framework for Reasoning-Driven Pedagogical Visualization. arXiv.
6. Building pipelines for educational data using AI and multimodal analytics: A “grey-box” approach10.1111/bjet.12854 ↗
7. Jones, K. (1988). A statistical interpretation of term specificity and its application in retrieval. In Document Retrieval Systems, Taylor Graham Publishing.
8. Experimentation as a way of life: Okapi at TREC10.1016/S0306-4573(99)00046-1 ↗
9. Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. (2023, January 2–6). MTEB: Massive Text Embedding Benchmark. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (ACL), Dubrovnik, Croatia.10.18653/v1/2023.eacl-main.148 ↗
10. Reimers, N., and Gurevych, I. (2019, January 3–7). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China.10.18653/v1/D19-1410 ↗
11. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W. (2020, January 16–20). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online Event.10.18653/v1/2020.emnlp-main.550 ↗
12. Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training. arXiv.
13. Khattab, O., and Zaharia, M. (2020, January 25–30). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’20), Virtual Event.10.1145/3397271.3401075 ↗
14. Bakir, G., Hofman, T., Sch¨olkopf, B., Smola, A., and Taskar, B. (2006). A Tutorial on Energy-Based Learning. Predicting Structured Data, MIT Press.10.7551/mitpress/7443.001.0001 ↗
15. Radford, A., Kim, J., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning (ICML), Virtual.
16. Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. (2022, January 18–24). Flava: A foundational language and vision alignment model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01519 ↗
17. Kwon, G., Cai, Z., Ravichran, A., Bas, E., Bhotika, R., and Soatto, S. (2023, January 1–5). Masked vision and language modeling for multi-modal representation learning. Proceedings of the Eleventh International Conference on Learning Representations (ICLR), Kigali, Rwanda.
18. Rombach, R., Blattman, A., Lorenz, D., Esser, P., and Ommer, B. (2022, January 18–24). High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01042 ↗
19. Photorealistic text-to-image diffusion models with deep language understanding
20. Yu, J., Xu, Y., Koh, J., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., and Ayan, B. (2022). Scaling autoregressive models for content-rich text-to-image generation. arXiv.
21. Multimodal few-shot learning with frozen language models
22. Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. (2023). MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv.
23. Li, J., Li, D., Savarese, S., and Hoi, S. (2023, January 23–29). BLIP-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. Proceedings of the 40th International Conference on Machine Learning (ICML), Honolulu, HI, USA.
24. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. (2023). Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv.
25. Intelligent Agents: Theory and Practice10.1017/S0269888900008122 ↗
26. Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., and Zhou, E. (2023). The Rise and Potential of Large Language Model Based Agents: A Survey. arXiv.
27. Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N., Wiest, O., and Zhang, X. (2024). Large Language Model based Multi-Agents: A Survey of Progress and Challenges. arXiv.
28. Park, J., O’Brien, J., Cai, C., Morris, M., Liang, P., and Bernstein, M. (November, January 29). Generative Agents: Interactive Simulacra of Human Behavior. Proceedings of the 36th Annual ACM Symposium On User Interface Software And Technology (UIST ’23), San Francisco, CA, USA.
29. Zhu, X., Chen, Y., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., and Wang, X. (2023). Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory. arXiv.
30. Fu, Y., Peng, H., Khot, T., and Lapata, M. (2023). Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback. arXiv.
31. ChatGPT has Entered the Classroom: How LLMs Could Transform Education10.1038/d41586-023-03507-3 ↗
32. Yue, M., Mifdal, W., Zhang, Y., Suh, J., and Yao, Z. (2024). MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics Education. arXiv.
33. Zhang, Z., Zhang-li, D., Yu, J., Gong, L., Zhou, J., Liu, Z., Hou, L., and Li, J. (2024). Simulating Classroom Education with LLM-Empowered Agents. arXiv.
34. Leveraging the Potential of Large Language Models in Education Through Playful and Game-Based Learning10.1007/s10648-024-09868-z ↗
35. Effective Learning with a Personal AI Tutor: A Case Study10.1007/s10639-024-12888-5 ↗
36. Park, M., Kim, S., Lee, S., Kwon, S., and Kim, K. (2024, January 11–16). Empowering Personalized Learning through a Conversation-based Tutoring System with Student Modeling. Proceedings of the CHI EA ’24: Extended Abstracts of the CHI Conference on Human Factors In Computing Systems, Honolulu, HI, USA.10.1145/3613905.3651122 ↗
37. Exploring the Potential of an AI-based Chatbot (ChatGPT) in Enhancing English as a Foreign Language (EFL) Teaching: Perceptions of EFL Faculty Members10.1007/s10639-023-11917-z ↗
38. What if the Devil Is My Guardian Angel: ChatGPT as a Case Study of Using Chatbots in Education10.1186/s40561-023-00237-x ↗
39. Do You Have AI Dependency? The Roles of Academic Self-efficacy, Academic Stress, and Performance Expectations on Problematic AI Usage Behavior10.1186/s41239-024-00467-0 ↗
40. Siu, O., Lui, K., Huang, Y., Ng, T., and Yeung, W. (2024). An Efficient, Reliable and Valid Assessment for Affective States during Online Learning. Sci. Rep., 14.10.1038/s41598-024-66974-2 ↗
41. Amina, A., Ahmad, A., Raghad, A., Mohamed, E., and Said, S. (2024). Ethical Implications of Using ChatGPT in Educational Environments: A Comprehensive Review. Artificial Intelligence In Education: The Power And Dangers Of ChatGPT In The Classroom, Springer.
42. Davey, L. (2017). DK Children’s Encyclopedia: The Book That Explains Everything, Dorling Kindersley.
43. Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot, C., and Colombo, P. (2025, January 24–28). ColPali: Efficient Document Retrieval with Vision Language Models. Proceedings of the 13th International Conference on Learning Representations (ICLR), Singapore.
44. Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. (2023, January 2–6). Sigmoid Loss for Language Image Pre-Training. Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France.10.1109/ICCV51070.2023.01100 ↗
45. Hu, E., Shen, Y., Wallis, P., Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022, January 25–29). LoRA: Low-Rank Adaptation of Large Language Models. Proceedings of the 2022 International Conference on Learning Representations (ICLR), Virtually.
46. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv.
47. OpenAI (2025, September 14). Introducing GPT-5. Available online: https://openai.com/index/introducing-gpt-5/.