Interactive Environment-Aware Planning System and Dialogue for Social Robots in Early Childhood Education
Source abstract
Abstract
In this study, we propose an interactive environment-aware dialog and planning system for social robots in early childhood education, aimed at supporting the learning and social interaction of young children. The proposed architecture consists of three core modules. First, semantic simultaneous localization and mapping (SLAM) accurately perceives the environment by constructing a semantic scene representation that includes attributes such as position, size, color, purpose, and material of objects, as well as their positional relationships. Second, the automated planning system enables stable task execution even in changing environments through planning domain definition language (PDDL)-based planning and replanning capabilities. Third, the visual question answering module leverages scene graphs and SPARQL conversion of natural language queries to answer children’s questions and engage in context-based conversations. The experiment conducted in a real kindergarten classroom with children aged 6 to 7 years validated the accuracy of object recognition and attribute extraction for semantic SLAM, the task success rate of the automated planning system, and the natural language question answering performance of the visual question answering (VQA) module.The experimental results confirmed the proposed system’s potential to support natural social interaction with children and its applicability as an educational tool.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Automated Systems · Social Robot Interaction and HRI
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronics Engineering, Chosun University, Gwangju 61452, Republic of Korea
Funding
Chosun University
References (51)
Show source-provided references
1. Physical activity and cognitive functioning of children: A systematic review10.3390/ijerph15040800 ↗
2. Object exploration is facilitated by the physical and social environment in center-based child care10.1111/cdev.14161 ↗
3. Environmental contributions to cognitive development: The role of cognitive stimulation10.1016/j.dr.2024.101135 ↗
4. The promotion of self-directed learning in pre-school: Reflection on teachers’ professional practice10.18844/cjes.v16i5.6351 ↗
5. Dore, R.A., and Dynia, J.M. (2020). Technology and media use in preschool classrooms: Prevalence, purposes, and contexts. Front. Educ., 5.10.3389/feduc.2020.600305 ↗
6. Social interactions in HRI: The robot view10.1109/TSMCC.2004.826268 ↗
7. Kanda, T., Shimada, M., and Koizumi, S. (2012, January 5–8). Children learning with a social robot. Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction, Boston, MA, USA.10.1145/2157689.2157809 ↗
8. Preschool children’s engagement with a social robot compared to a human instructor10.1016/j.ecresq.2023.07.010 ↗
9. The use of social robots in classrooms: A review of field-based studies10.1016/j.edurev.2021.100388 ↗
10. Lampropoulos, G. (2025). Social robots in education: Current trends and future perspectives. Information, 16.10.3390/info16010029 ↗
11. Tablet computers in early science education: Enriching teacher–child interactions10.1007/s10643-024-01766-x ↗
12. “Robot, tell me a tale!”A social robot as tool for teachers in kindergarten10.1075/is.18024.con ↗
13. Keren, G., Ben-David, A., and Fridin, M. (2012, January 7–12). Kindergarten assistive robotics (KAR) as a tool for spatial cognition development in pre-school education. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal.10.1109/IROS.2012.6385645 ↗
14. De Wit, J., Schodde, T., Willemsen, B., Bergmann, K., De Haas, M., Kopp, S., Krahmer, E., and Vogt, P. (2018, January 5–8). The effect of a robot’s gestures and adaptive tutoring on children’s acquisition of second language vocabularies. Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, Chicago, IL, USA.10.1145/3171221.3171277 ↗
15. Wu, X.E., and Ko, J. (2024). Peer interactions during storybook reading on children’s knowledge construction: An experimental study on K2 and K3 children. Front. Educ., 9.10.3389/feduc.2024.1253782 ↗
16. Cankaya, O., Rohatyn-Martin, N., Leach, J., Taylor, K., and Bulut, O. (2023). Preschool children’s loose parts play and the relationship to cognitive development: A review of the literature. J. Intell., 11.10.3390/jintelligence11080151 ↗
17. Alqobali, R., Alnasser, R., Rashidi, A., Alshmrani, M., and Alhmiedat, T. (2024). A real-time semantic map production system for indoor robot navigation. Sensors, 24.10.3390/s24206691 ↗
18. Zheng, C., Zhang, P., and Li, Y. (2025). Semantic SLAM system for mobile robots based on large visual model in complex environments. Sci. Rep., 15.10.1038/s41598-025-90340-5 ↗
19. Jiang, Y., Wu, Y., and Zhao, B. (2025). Enhancing SLAM algorithm with Top-K optimization and semantic descriptors. Sci. Rep., 15.10.1038/s41598-025-90968-3 ↗
20. A knowledge-based task planning approach for robot multi-task manipulation10.1007/s40747-023-01155-8 ↗
21. Golluccio, G., Di Vito, D., Marino, A., Bria, A., and Antonelli, G. (2021, January 6–8). Task-motion planning via tree-based Q-learning approach for robotic object displacement in cluttered spaces. Proceedings of the 18th International Conference on Informatics in Control, Automation and Robotics (ICINCO), Online.10.5220/0010542600002994 ↗
22. Chalvatzaki, G., Younes, A., Nandha, D., Le, A.T., Ribeiro, L.F.R., and Gurevych, I. (2023). Learning to reason over scene graphs: A case study of finetuning GPT-2 into a robot language model for grounded task plan. Front. Robot., 10.10.3389/frobt.2023.1221739 ↗
23. Automatic extension of a symbolic mobile manipulation skill set10.1016/j.robot.2023.104428 ↗
24. Liu, R., Wan, G., Jiang, M., Chen, H., and Zeng, P. (2024). Autonomous robot task execution in flexible manufacturing: Integrating PDDL and behavior trees in ARIAC 2023. Biomimetics, 9.10.3390/biomimetics9100612 ↗
25. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., and Parikh, D. (2015, January 7–13). VQA: Visual question answering. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.279 ↗
26. Visual question answering: A survey of methods and datasets10.1016/j.cviu.2017.05.001 ↗
27. Kafle, K., and Kanan, C. (2017, January 22–29). An analysis of visual question answering algorithms. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.217 ↗
28. Exploring models and data for image question answering
29. Ask your neurons: A deep learning approach to visual question answering10.1007/s11263-017-1038-2 ↗
30. Neural-symbolic VQA: Disentangling reasoning from vision and language understanding
31. A neuro-symbolic ASP pipeline for visual question answering10.1017/S1471068422000229 ↗
32. Amizadeh, S., Palangi, H., Polozov, A., Huang, Y., and Koishida, K. (2020, January 12–18). Neuro-symbolic visual reasoning: Disentangling. Proceedings of the International Conference on Machine Learning (ICML), Virtual Event.
33. Exploring the multifaceted roles of social robots in early childhood literacy lessons: Insights from a Maltese classroom10.1007/s12369-025-01290-x ↗
34. Johnson, A., Martin, A., Quintero, M., Bailey, A., and Alwan, A. (2022, January 28–31). Can social robots effectively elicit curiosity in STEM topics from K-1 students during oral assessments?. Proceedings of the 2022 IEEE Global Engineering Education Conference (EDUCON), Tunis, Tunisia.10.1109/EDUCON52537.2022.9766662 ↗
35. Mei, L., and Xu, P. (2024). Path planning for robots combined with zero-shot and hierarchical reinforcement learning in novel environments. Actuators, 13.10.3390/act13110458 ↗
36. Zhao, B., Wu, Y., Wu, C., and Sun, R. (2025). Deep reinforcement learning trajectory planning for robotic manipulator based on simulation-efficient training. Sci. Rep., 15.10.1038/s41598-025-93175-2 ↗
37. Deep reinforcement learning for path planning of autonomous mobile robots in complicated environments10.1007/s40747-025-01906-9 ↗
38. Cashmore, M., Fox, M., Long, D., Magazzeni, D., Ridder, B., Carrera, A., Palomeras, N., Hurtos, N., and Carreras, M. (2015, January 7–11). Rosplan: Planning in the robot operating system. Proceedings of the 25th International Conference on Automated Planning and Scheduling (ICAPS), Jerusalem, Israel.10.1609/icaps.v25i1.13699 ↗
39. Moon, J., and Lee, B.-H. (2019). PDDL planning with natural language-based scene understanding for UAV-UGV cooperation. Appl. Sci., 9.10.3390/app9183789 ↗
40. Concept for the automated adaption of abstract planning domains for specific application cases in skills-based industrial robotics10.1007/s10845-023-02211-3 ↗
41. Hierarchical online automated planning for a flexible manufacturing system10.1016/j.rcim.2024.102807 ↗
42. Mascharka, D., Tran, P., Soklaski, R., and Majumdar, A. (2018, January 18–23). Transparency by design: Closing the gap between performance and interpretability in visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00519 ↗
43. Moon, J. (2023). Symmetric graph-based visual question answering using neuro-symbolic approach. Symmetry, 15.10.3390/sym15091713 ↗
44. Peña-Narvaez, J.D., Martín, F., Guerrero, J.M., and Pérez-Rodríguez, R. (2023). A visual questioning answering approach to enhance robot localization in indoor environments. Front. Neurorobot., 17.10.3389/fnbot.2023.1290584 ↗
45. Transformer-based vision-language alignment for robot navigation and question answering10.1016/j.inffus.2024.102351 ↗
46. EduVQA: A multimodal visual question answering framework for smart education10.1016/j.aej.2025.03.005 ↗
47. Sankalprajan, P., Sharma, T., Perur, H.D., and Pagala, P.S. (2020, January 5–7). Comparative analysis of ROS based 2D and 3D SLAM algorithms for Autonomous Ground Vehicles. Proceedings of the 2020 International Conference for Emerging Technology (INCET), Belgaum, India.10.1109/INCET49848.2020.9154101 ↗
48. Tsai, H., Riesa, J., Johnson, M., Arivazhagan, N., Li, X., and Archer, A. (2019). Small and practical BERT models for sequence labeling. arXiv.10.18653/v1/D19-1374 ↗
49. Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Zitnick, C.L., and Girshick, R. (2017, January 21–26). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.215 ↗
50. MJ, A.K., Babu, A.V., Damodaran, S., James, R.K., Murshid, M., and Warrier, T.S. (2024, January 24–26). ROS2-Powered Autonomous Navigation for TurtleBot3: Integrating Nav2 Stack in Gazebo, RViz and Real-World Environments. Proceedings of the 2024 IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (SPICES), Kollam, India.
51. Shakhovska, N., Basystiuk, O., and Shakhovska, K. (June, January 31). Development of the Speech-to-Text Chatbot Interface Based on Google API. Proceedings of the International Conference on Modern Machine Learning Technologies (MoMLeT), Lviv, Ukraine.