Geometric Scene Formalization in Vision-Based Educational Sensing via Multimodal Large Models
Source abstract
Abstract
This paper studies geometric scene formalization in vision-based educational sensing environments, where textual conditions and geometric diagram images jointly constitute heterogeneous perceptual inputs. The goal is to convert multimodal sensed information into standardized formal representations for machine understandable educational analysis. Existing methods remain limited by unstable cross modal alignment, inadequate expression of geometric relational constraints, and insufficient verifiability of generated outputs. To overcome these challenges, a unified modeling framework is proposed based on multimodal large models with structure-aware prompting and verification feedback. A geometry-oriented structure prompt injection mechanism is first introduced to encode prior cues of geometric entities, relational patterns, and constraint dependencies, which enhances the intrinsic alignment among textual descriptions, visually sensed diagram regions, and formal symbolic representations. In addition, an external verification feedback strategy is employed to constrain and iteratively refine the initial outputs, thereby improving structural consistency, syntactic correctness, and target proposition accuracy. To support this task, a new vision-based multimodal geometry formalization dataset is further constructed for model training and evaluation. Extensive experiments show that the proposed method can more effectively accomplish the transformation from multimodal sensed educational inputs to executable formal expressions, while also demonstrating stronger robustness and reliability in complex visual conditions. These results indicate that the proposed framework offers a feasible solution for structured scene interpretation, automatic problem analysis, error diagnosis, and intelligent feedback in vision-based educational systems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multimodal Machine Learning Applications · Handwritten Text Recognition Techniques · Data Visualization and Analytics
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Computing Science & Technology, Guangzhou University, Guangzhou 510006, China
Funding
Guangzhou University
References (31)
Show source-provided references
1. Seo, M., Hajishirzi, H., Farhadi, A., Etzioni, O., and Malcolm, C. (2015, January 17–21). Solving geometry problems: Combining text and diagram interpretation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1171 ↗
2. Sachan, M., and Xing, E. (2017, January 3–4). Learning to solve geometry problems from natural language demonstrations in textbooks. Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), Vancouver, BC, Canada.10.18653/v1/S17-1029 ↗
3. Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., and Zhu, S.C. (2021, January 1–6). Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Virtual Event.10.18653/v1/2021.acl-long.528 ↗
4. Ning, M., Wang, Q.F., Huang, K., and Huang, X. (November, January 29). A symbolic characters aware model for solving geometry problems. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
5. Wu, W., Zhang, L., Liu, J., Tang, X., Wang, Y., Wang, S., and Wang, Q. (2024, January 16–22). E-gps: Explainable geometry problem solving via top-down solver and bottom-up generator. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.01312 ↗
6. Zhang, X., Zhu, N., Qin, C., Li, Y., Zeng, Z., and Leng, T. (2024, January 14). Formal representation and solution of plane geometric problems. Proceedings of the 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24, Vancouver, BC, Canada.
7. Chen, J., Tang, J., Qin, J., Liang, X., Liu, L., Xing, E., and Lin, L. (2021, January 1–6). Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. Proceedings of the Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online Event.10.18653/v1/2021.findings-acl.46 ↗
8. Hao, Y., Zhang, M., Yin, F., and Huang, L.L. (2022). PGDP5K: A diagram parsing dataset for plane geometry problems. Proceedings of the 2022 26th International Conference on Pattern Recognition (ICPR), IEEE.10.1109/ICPR56361.2022.9956397 ↗
9. Zhang, M.L., Yin, F., and Liu, C.L. (2023). A multi-modal neural geometric solver with textual clauses parsed from diagram. arXiv.10.24963/ijcai.2023/376 ↗
10. Li, Z.Z., Zhang, M.L., Yin, F., and Liu, C.L. (2024, January 11–16). LANS: A layout-aware neural solver for plane geometry problem. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand.10.18653/v1/2024.findings-acl.153 ↗
11. Gao, J., Pi, R., Zhang, J., Ye, J., Zhong, W., Wang, Y., Hong, L., Han, J., Xu, H., and Li, Z. (2023). G-llava: Solving geometric problem with multi-modal large language model. arXiv.
12. Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.W., Galley, M., and Gao, J. (2023). Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv.
13. Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.W., and Qiao, Y. (2024). Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?. Proceedings of the European Conference on Computer Vision, Springer.10.1007/978-3-031-73242-3_10 ↗
14. Shi, W., Hu, Z., Bin, Y., Liu, J., Yang, Y., Ng, S.K., Bing, L., and Lee, R.K.W. (2024, January 12–16). Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, FL, USA.10.18653/v1/2024.findings-emnlp.268 ↗
15. Han, J.M., Rute, J., Wu, Y., Ayers, E.W., and Polu, S. (2021). Proof artifact co-training for theorem proving with language models. arXiv.
16. Zheng, K., Han, J.M., and Polu, S. (2021). Minif2f: A cross-system benchmark for formal olympiad-level mathematics. arXiv.
17. Autoformalization with large language models10.52202/068431-2344 ↗
18. Azerbayev, Z., Piotrowski, B., Schoelkopf, H., Ayers, E.W., Radev, D., and Avigad, J. (2023). Proofnet: Autoformalizing and formally proving undergraduate-level mathematics. arXiv.
19. Leandojo: Theorem proving with retrieval-augmented language models10.52202/075280-0944 ↗
20. Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M.D., McAleer, S., Jiang, A.Q., Deng, J., Biderman, S., and Welleck, S. (2023). Llemma: An open language model for mathematics. arXiv.
21. Lu, J., Wan, Y., Liu, Z., Huang, Y., Xiong, J., Liu, C., Shen, J., Jin, H., Zhang, J., and Wang, H. (2024). Process-driven autoformalization in lean 4. arXiv.
22. Wang, R., Zhang, J., Jia, Y., Pan, R., Diao, S., Pi, R., and Zhang, T. (2024, January 12–16). Theoremllama: Transforming general-purpose llms into lean4 experts. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.10.18653/v1/2024.emnlp-main.667 ↗
23. Xin, H., Guo, D., Shao, Z., Ren, Z., Zhu, Q., Liu, B., Ruan, C., Li, W., and Liang, X. (2024). Deepseek-prover: Advancing theorem proving in llms through large-scale synthetic data. arXiv.
24. Patel, M., Bhattacharyya, R., Lu, T., Mehta, A., Voss, N., Norouzi, N., and Ranade, G. (2026). LeanTutor: Towards a Verified AI Mathematical Proof Tutor. arXiv.10.1609/aaai.v40i47.41514 ↗
25. Tan, Z., Yang, X., Qin, L., Yang, M., Zhang, C., and Li, H. (2024). Evalalign: Supervised fine-tuning multimodal llms with human-aligned data for evaluating text-to-image models. arXiv.
26. Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Dan, Y., Zhao, C., Xu, G., Li, C., and Tian, J. (2023). mplug-docowl: Modularized multimodal large language model for document understanding. arXiv.
27. Yu, Y.Q., Liao, M., Wu, J., Liao, Y., Zheng, X., and Zeng, W. (2024). Texthawk: Exploring efficient fine-grained perception of multimodal large language models. arXiv.
28. Zhou, B., Hu, Y., Weng, X., Jia, J., Luo, J., Liu, X., Wu, J., and Huang, L. (2024). Tinyllava: A framework of small-scale large multimodal models. arXiv.
29. Qiao, L., Wang, H., Sugiura, K., Liu, K., and Hu, J. (2025). MMtuning: An Advanced Multi-adapter Framework for Efficient Multimodal Large Language Models Fine-Tuning. Proceedings of the International Conference on Knowledge Science, Engineering and Management, Springer.10.1007/978-981-95-3055-7_5 ↗
30. Zhao, Y., Huang, J., Hu, J., Wang, X., Mao, Y., Zhang, D., Jiang, Z., Wu, Z., Ai, B., and Wang, A. (March, January 25). Swift: A scalable lightweight infrastructure for fine-tuning. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA.
31. Jiang, T., Song, M., Zhang, Z., Huang, H., Deng, W., Sun, F., Zhang, Q., Wang, D., and Zhuang, F. (2024). E5-v: Universal embeddings with multimodal large language models. arXiv.10.18653/v1/2024.findings-emnlp.181 ↗