Exploration of Stability Judgments: Assessing Multimodal LLMs in Game-Inspired Physical Reasoning Tasks
Source abstract
Abstract
This study extends our previous investigation into whether multimodal large language models (MLLMs) can reason about physical reasoning, using a game environment as the testbed. Stability served as a foundational scenario to probe model understanding of physical reasoning. We evaluated twelve models, combining those from the earlier study with six additional open-weight models, across three tasks designed to capture different aspects of reasoning. Human participants were included as a reference point, consistently achieving the highest accuracy, underscoring the gap between model and human performance. Among MLLMs, the GPT series continued to perform strongly, with GPT-4o showing reliable results in image-based tasks, while the Qwen2.5-VL series reached the highest overall scores in this extended study and in some cases surpassed commercial counterparts. Simpler binary tasks yielded balanced performance across modalities, suggesting that models can capture certain basic aspects of reasoning, whereas more complex multiple-choice tasks led to sharp declines in accuracy. Structured inputs such as XML improved results in the prediction task, where Qwen2.5-VL outperformed GPT variants in our earlier work. These findings demonstrate progress in scaling and modality design for physical reasoning, while reaffirming that human participants remain superior across all tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Educational Games and Gamification · Intelligent Tutoring Systems and Adaptive Learning · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Information Science and Engineering, Ritsumeikan University, 2-150 Iwakura-cho, Ibaraki 567-8570, Osaka, Japan
Center for Information and Neural Networks, National Institute of Information and Communications Technology, Suita 565-0871, Osaka, Japan
College of Information Science and Engineering, Ritsumeikan University, 2-150 Iwakura-cho, Ibaraki 567-8570, Osaka, Japan
Alumni of College of Engineering, Electrical and Computer Engineering Department, Carnegie Mellon University, Pittsburgh, PA 15213, USA
Faculty of Computer Science, Universitas Brawijaya, Malang 65145, Indonesia
References (27)
Show source-provided references
1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
2. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Tang, J., and Zhong, H. (2025). Qwen2. 5-vl technical report. arXiv.
3. Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., and Rouillard, L. (2025). Gemma 3 technical report. arXiv.
4. Liu, H., Li, C., Li, Y., and Lee, Y.J. (2024, January 17–21). Improved baselines with visual instruction tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02484 ↗
5. Jassim, S., Holubar, M., Richter, A., Wolff, C., Ohmer, X., and Bruni, E. (2023). Grasp: A novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models. arXiv.
6. Voudouris, K., Donnelly, N., Rutar, D., Burnell, R., Burden, J., Hernández-Orallo, J., and Cheke, L.G. (2022, January 25). Evaluating object permanence in embodied agents using the Animal-AI environment. Proceedings of the EBeM’22: Workshop on AI Evaluation Beyond Metrics, Vienna, Austria.
7. Dewantoro, M.F., Abdullah, F., Xia, Y., Khan, I., Thawonmas, R., and Ouyang, W. (2025, January 26–29). Can Multimodal LLMs Reason About Stability? An Exploratory Study with Insights from the LLMs4PCG Challenge. Proceedings of the 2025 IEEE Conference on Games (CoG), Lisboa, Portugal.10.1109/CoG64752.2025.11114183 ↗
8. Zhu, E., Liu, Y., Zhang, Z., Li, X., Zhou, J., Yu, X., Huang, M., and Wang, H. (2025, January 24–28). MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Science. Proceedings of the Thirteenth International Conference on Learning Representations, Singapore.
9. Visual cognition in multimodal large language models10.1038/s42256-024-00963-y ↗
10. Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv.
11. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., and Metzler, D. (2022). Emergent Abilities of Large Language Models. arXiv.
12. Elangovan, A., Liu, L., Xu, L., Bodapati, S.B., and Roth, D. (2024, January 11–16). ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand.10.18653/v1/2024.acl-long.63 ↗
13. Verghese, M., Chen, B., Eghbalzadeh, H., Nagarajan, T., and Desai, R. (March, January 26). User-in-the-Loop Evaluation of Multimodal LLMs for Activity Assistance. Proceedings of the 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Tucson, AZ, USA.10.1109/WACV61041.2025.00119 ↗
14. Chen, D., Chen, R., Zhang, S., Liu, Y., Wang, Y., Zhou, H., Zhang, Q., Zhou, P., Wan, Y., and Sun, L. (2024, January 21–27). MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark. Proceedings of the International Conference on Machine Learning, Vienna, Austria.
15. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models
16. Hao, D., Wang, Q., Guo, L., Jiang, J., and Liu, J. (2024, January 12–16). Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answering. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.10.18653/v1/2024.emnlp-main.110 ↗
17. Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. (2024, January 14–19). Extending Large Language Models for Speech and Audio Captioning. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10446343 ↗
18. Li, M., Do, C.T., Keizer, S., Farag, Y., Stoyanchev, S., and Doddipatla, R. (2024, January 2–5). WHISMA: A Speech-LLM to Perform Zero-Shot Spoken Language Understanding. Proceedings of the 2024 IEEE Spoken Language Technology Workshop (SLT), Macao, China.10.1109/SLT61566.2024.10832156 ↗
19. Feng, Z., Dou, J., Ni, Z., and Guan, H. (2025, January 16–18). Lightweight Design of Robotic Arm Based on Multimodal Large Language Models. Proceedings of the 2025 5th International Conference on Computer, Control and Robotics (ICCCR), Hangzhou, China.10.1109/ICCCR65461.2025.11072572 ↗
20. Zhao, Q., Lu, Y., Kim, M.J., Fu, Z., Zhang, Z., Wu, Y., Li, Z., Ma, Q., Han, S., and Finn, C. (2025, January 10–17). CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models. Proceedings of the 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR52734.2025.00166 ↗
21. Learn to explain: Multimodal reasoning via thought chains for science question answering
22. Raza, S., Narayanan, A., Khazaie, V.R., Vayani, A., Chettiar, M.S., Singh, A., Shah, M., and Pandya, D. (2025). Humanibench: A human-centric framework for large multimodal models evaluation. arXiv.
23. Abdullah, F., Paliyawan, P., Thawonmas, R., Harada, T., and Bachtiar, F.A. (2019, January 20–23). An Angry Birds Level Generator with Rube Goldberg Machine Mechanisms. Proceedings of the 2019 IEEE Conference on Games (CoG), London, UK.10.1109/CIG.2019.8847996 ↗
24. Taveekitworachai, P., Abdullah, F., and Thawonmas, R. (2024, January 12–16). Null-Shot Prompting: Rethinking Prompting Large Language Models With Hallucination. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.10.18653/v1/2024.emnlp-main.740 ↗
25. Oh, A.H., Agarwal, A., Belgrave, D., and Cho, K. (2022). Chain of Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems, Proceedings of the Conference and Workshop on Neural Information Processing Systems, New Orleans, LA, USA, 28 November 2022–9 December 2022, Curran Associates, Inc.
26. Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., and Garnett, R. (2019). PHYRE: A New Benchmark for Physical Reasoning. Advances in Neural Information Processing Systems, Proceedings of the Conference and Workshop on Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019, Curran Associates, Inc.
27. Li, S., Wu, K., Zhang, C., and Zhu, Y. (2024, January 7–11). I-PHYRE: Interactive Physical Reasoning. Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria.