DL-ReasonSuite: A Benchmark for Evaluating Description Logic Reasoning in Large Language Models
Source abstract
Abstract
Large language models (LLMs) have shown remarkable progress in general reasoning and understanding, but their ability to perform formal logical reasoning remains under-explored. In this paper, we introduce DLReasonSuite, a novel benchmark designed to rigorously evaluate LLMs on reasoning tasks grounded in Description Logic (DL). DL-ReasonSuite comprises 4740 tasks spanning seven distinct task types and organized into three reasoning tracks: (1) DLCore, covering fundamental ontology reasoning tasks (consistency checking, subsumption, and instance checking); (2) DLQuery, focusing on answering entailment-aware SPARQL queries; and (3) DLBridge, bridging natural language and formal logic (bidirectional NL ↔ OWL translation and tool-augmented entailment resolution). We detail the methodology for designing and implementing this benchmark, including task construction, automatic evaluation metrics and validation using reliable OWL reasoners. Then, we present an empirical evaluation of five leading reasoning LLMs as stateofart models: Kimi k1.5, LlamaNemotron Ultra, DeepSeekR1, Phi4 Reasoning Plus, and Phi4 Reasoning on the full suite of tasks. Our results reveal significant variability in LLM performance on formal reasoning was observed. While the best model, Phi4 Reasoning Plus, achieves an overall accuracy of 85% and excels especially in tool-augmented tasks, other models struggle notably with complex query reasoning for DL and precise OWL translation. We analyze the strengths and weaknesses of each model across different DL metrics and task categories, providing insights into current limitations of LLM reasoning such as handling SPARQL queries and maintaining logical consistency and the benefits of neuro-symbolic techniques. DL-ReasonSuite is a comprehensive framework for assessing and advancing LLMs’ Description Logic reasoning capabilities aiming to bridge the gap between natural language understanding and formal knowledge representation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Advanced Graph Neural Networks
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Engineering Department, Faculty of Engineering and Architecture, Izmir Bakircay University, 35665 Izmir, Turkey
References (42)
Show source-provided references
1. Bender, E.M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big?. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21), Virtual Event, 3–10 March 2021, ACM.10.1145/3442188.3445922 ↗
2. Gendron, G., Bao, Q., Witbrock, M., and Dobbie, G. (2023). Large language models are not robust reasoners. arXiv.
3. McCoy, R.T., Pavlick, E., and Linzen, T. (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019), Florence, Italy, 28 July–2 August 2019, Association for Computational Linguistics.10.18653/v1/P19-1334 ↗
4. Baader, F., Calvanese, D., McGuinness, D.L., Nardi, D., and Patel-Schneider, P.F. (2003). The Description Logic Handbook: Theory, Implementation and Applications, Cambridge University Press.
5. OWL 2: The next step for OWL10.1016/j.websem.2008.05.001 ↗
6. Horrocks, I., Kutz, O., and Sattler, U. (2006, January 2–5). The even more irresistible SROIQ. Proceedings of the 10th International Conference on Principles of Knowledge Representation and Reasoning (KR 2006), Lake, UK.
7. Pellet: A practical OWL-DL reasoner10.1016/j.websem.2007.03.004 ↗
8. HermiT: An OWL 2 reasoner10.1007/s10817-014-9305-1 ↗
9. Parsia, B., Matentzoglu, N., Gonçalves, R.S., Glimm, B., and Steigmiller, A. (2015, January 11). The OWL Reasoner Evaluation (ORE) 2015 Competition Report. Proceedings of the 11th International Workshop on Scalable Semantic Web Knowledge Base Systems, Bethlehem, PA, USA.
10. LUBM: A benchmark for OWL knowledge base systems10.1016/j.websem.2005.06.005 ↗
11. Ma, L., Yang, Y., Qiu, Z., Xie, G., Pan, Y., and Liu, S. (2006). Towards a complete OWL ontology benchmark. Proceedings of the 3rd European Semantic Web Conference (ESWC 2006), Budva, Montenegro, 11–14 June 2006, Springer.10.1007/11762256_12 ↗
12. Singh, G., Kumar, A., Bhagat, K., Bhatia, S., and Mutharaju, R. (2020, January 2–6). OWL2Bench: Towards a customizable benchmark for OWL 2 reasoners. Proceedings of the 19th International Semantic Web Conference (ISWC 2020), Athens, Greece.10.1007/978-3-030-62466-8_6 ↗
13. Matentzoglu, N., Bail, S., and Parsia, B. (2013, January 26–27). A corpus of OWL DL ontologies. Proceedings of the OWL: Experiences and Directions Workshop (OWLED 2013), Montpellier, France.
14. Teyou, L.M.K., Friedrichs, L., Kouagou, N.J., Demir, C., Mahmood, Y., Heindorf, S., and Ngonga Ngomo, A.C. (2025, January 24–28). Neural Description Logic Reasoning over Incomplete Knowledge Bases. Proceedings of the Thirteenth International Conference on Learning Representations ICLR 2025, Singapore. Available online: https://openreview.net/forum?id=4qRCiEZGKd.
15. Li, Z., Huang, M., Zhong, Y., and Qin, Y. (2022). A Description Logic Based Ontology for Knowledge Representation in Process Planning for Laser Powder Bed Fusion. Appl. Sci., 12, Available online: https://www.mdpi.com/2076-3417/12/9/4612.10.3390/app12094612 ↗
16. Borgwardt, S., De Bortoli, F., and Koopmann, P. (2024, January 18–21). The Precise Complexity of Reasoning in ALC with D-Admissible Concrete Domains. Proceedings of the 37th International Workshop on Description Logics (DL 2024), Bergen, Norway.
17. Lutz, C., and Schulze, L. (2024, January 2–8). Description Logics with Abstraction and Refinement: From ALC to EL. Proceedings of the 21st International Conference on Principles of Knowledge Representation and Reasoning, Hanoi, Vietnam.10.24963/kr.2024/51 ↗
18. Manière, Q., and Przybyłko, M. (March, January 25). Spectra of Cardinality Queries over Description Logic Knowledge Bases. Proceedings of the AAAI Conference on Artificial Intelligence 2025, Philadelphia, PA, USA.10.1609/aaai.v39i14.33652 ↗
19. Zhang, H., Jiang, G., and Quan, D. (March, January 25). A Theory of Formalisms for Representing Knowledge. Proceedings of the AAAI Conference on Artificial Intelligence 2025, Philadelphia, PA, USA.10.1609/aaai.v39i14.33674 ↗
20. Di Stefano, F., Manière, Q., Ortiz, M., and Šimkus, M. (2025). Minimal model reasoning in description logics: Don’t try this at home!. arXiv.10.24963/kr.2025/27 ↗
21. Morawska, B., and Marzec, D. (2025, January 11–17). Solving Unification in the Description Logic FL⊥. Proceedings of the 22nd International Conference on Principles of Knowledge Representation and Reasoning, Melbourne, Australia.10.24963/kr.2025/46 ↗
22. Di Stefano, F., and Šimkus, M. (2024, January 20–27). Stable Model Semantics for Description Logic Terminologies. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i9.28917 ↗
23. Casini, G., Haldimann, J., and Meyer, T. (2025, January 11–17). Reasoning in defeasible description logics with system W and lexicographic inference. Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, Melbourne, Australia.10.24963/kr.2025/22 ↗
24. Koopmann, P. (2025). Explaining Reasoning Results for Description Logic Ontologies. Joint Proceedings of the 20th and 21st Reasoning Web Summer Schools (RW 2024 & RW 2025), Schloss Dagstuhl–Leibniz-Zentrum für Informatik.
25. Linear reasoning: A new form of the Herbrand–Gentzen theorem10.2307/2963593 ↗
26. Koopmann, P., Mahmood, Y., Ngomo, A.C.N., and Tiwari, B. (2025). Can You Tell the Difference? Contrastive Explanations for ABox Entailments. arXiv.
27. Bridging the gap: Neuro-Symbolic Computing for advanced AI applications in construction10.1007/s42524-023-0266-0 ↗
28. Li, H., Cohan, A., Witbrock, M.J., Yang, M., Liu, F., Lin, Z., van Benthem, J., and Clark, P. (2026, January 23–27). Workshop on Logical Reasoning of Large Language Models. Proceedings of the ICLR 2026 Workshop Proposals, The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil.
29. Link, V., Lohmann, S., and Haag, F. (2016). OntoBench: Generating custom OWL 2 benchmark ontologies. Proceedings of the International Semantic Web Conference (ISWC), Kobe, Japan, 17–21 October 2016, Springer.10.1007/978-3-319-46547-0_13 ↗
30. Fang, M., Deng, S., Zhang, Y., Shi, Z., Chen, L., Pechenizkiy, M., and Wang, J. (2024, January 20–27). Large Language Models Are Neurosymbolic Reasoners. Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI 2024), Vancouver, BC, Canada.10.1609/aaai.v38i16.29754 ↗
31. Language Models are Few-Shot Learners
32. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., and Zhou, D. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv.
33. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023, January 1–5). ReAct: Synergizing Reasoning and Acting in Language Models. Proceedings of the 11th International Conference on Learning Representations (ICLR 2023), Kigali, Rwanda.
34. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv.
35. Pan, L., Albalak, A., Wang, X., and Wang, W.Y. (2023). Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning. Findings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), Association for Computational Linguistics.10.18653/v1/2023.findings-emnlp.248 ↗
36. Olausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J.B., and Levy, R. (2023, January 6–10). 1LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), Singapore.10.18653/v1/2023.emnlp-main.313 ↗
37. Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. (2020, January 11–17). LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning. Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI 2020), Yokohama, Japan.10.24963/ijcai.2020/501 ↗
38. Tafjord, O., Dalvi, B., and Clark, P. (2021, January 7–11). ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), Virtual Event.10.18653/v1/2021.findings-acl.317 ↗
39. Yu, W., Jiang, Z., Dong, Y., and Feng, J. (2020). ReClor: A reading comprehension dataset requiring logical reasoning. arXiv.
40. Parmar, M., Patel, N., Varshney, N., Nakamura, M., Luo, M., Mashetty, S., Mitra, A., and Baral, C. (2024). LogicBench: Towards systematic evaluation of logical reasoning ability of large language models. arXiv.10.18653/v1/2024.acl-long.739 ↗
41. Chung, T.T., Liu, L., Yu, M., and Yeung, D.Y. (2025). DivLogicEval: A framework for benchmarking logical reasoning evaluation in large language models. Findings of the Association for Computational Linguistics: EMNLP, Association for Computational Linguistics.10.18653/v1/2025.findings-emnlp.47 ↗
42. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2023, January 1–5). Self-consistency improves chain of thought reasoning in language models. Proceedings of the 11th International Conference on Learning Representations (ICLR), Kigali, Rwanda.