Large Language Models Imitate Logical Reasoning, but at what Cost?

2025-09-16

AI Summary

This study systematically evaluates the evolution of logical reasoning capabilities in state-of-the-art large language models (LLMs) over an 18-month period (2023–2025), focusing on accuracy and fidelity to reasoning strategies under in-context learning on the PrOntoQA benchmark. To address computational inefficiency in end-to-end LLM-based reasoning, we propose a lightweight neuro-symbolic architecture: questions are first normalized into first-order logic and then verified for satisfiability using the Z3 SMT solver—enabling near-optimal accuracy with models under 15B parameters. This approach substantially reduces FLOPs and generated token count compared to purely end-to-end methods. Empirical analysis reveals that performance gains from 2023 to 2024 stemmed primarily from implicit chain-of-thought emergence, while advances from 2024 to 2025 were driven by explicit “reasoning models.” Our core contribution is the empirical validation of neuro-symbolic integration as a viable path to high-accuracy logical reasoning with significantly reduced computational overhead.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present a longitudinal study which evaluates the reasoning capability of frontier Large Language Models over an eighteen month period. We measured the accuracy of three leading models from December 2023, September 2024 and June 2025 on true or false questions from the PrOntoQA dataset and their faithfulness to reasoning strategies provided through in-context learning. The improvement in performance from 2023 to 2024 can be attributed to hidden Chain of Thought prompting. The introduction of thinking models allowed for significant improvement in model performance between 2024 and 2025. We then present a neuro-symbolic architecture which uses LLMs of less than 15 billion parameters to translate the problems into a standardised form. We then parse the standardised forms of the problems into a program to be solved by Z3, an SMT solver, to determine the satisfiability of the query. We report the number of prompt and completion tokens as well as the computational cost in FLOPs for open source models. The neuro-symbolic approach significantly reduces the computational cost while maintaining near perfect performance. The common approximation that the number of inference FLOPs is double the product of the active parameters and total tokens was accurate within 10% for all experiments.

Problem

• Evaluating reasoning capability of Large Language Models over time
• Assessing faithfulness to reasoning strategies in PrOntoQA dataset
• Developing neuro-symbolic architecture to reduce computational costs

Innovation

• Hidden Chain of Thought prompting improves reasoning
• Thinking models enhance model performance significantly
• Neuro-symbolic architecture with Z3 solver reduces cost