Understanding Syllogistic Reasoning in LLMs from Formal and Natural Language Perspectives

2025-12-14

AI Summary

It remains unclear whether large language models (LLMs) possess genuine formal logical reasoning capabilities—particularly in syllogistic inference—or merely emulate human intuitive reasoning through statistical pattern matching. Method: We introduce the first unified evaluation framework that jointly assesses symbolic logical validity and natural language comprehension, benchmarking 14 state-of-the-art LLMs on a standardized syllogism test suite. Contribution/Results: Syllogistic reasoning is not a universally emergent capability across LLMs; rather, performance varies significantly. Notably, several models achieve 100% accuracy on symbolic syllogistic tasks, demonstrating behavior closely aligned with formal logic engines. This challenges the prevailing assumption that LLMs rely solely on surface-level statistical correlations to mimic reasoning. Our work provides critical empirical evidence and a methodological foundation for characterizing the nature of LLM reasoning and advancing trustworthy AI systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We study syllogistic reasoning in LLMs from the logical and natural language perspectives. In process, we explore fundamental reasoning capabilities of the LLMs and the direction this research is moving forward. To aid in our studies, we use 14 large language models and investigate their syllogistic reasoning capabilities in terms of symbolic inferences as well as natural language understanding. Even though this reasoning mechanism is not a uniform emergent property across LLMs, the perfect symbolic performances in certain models make us wonder whether LLMs are becoming more and more formal reasoning mechanisms, rather than making explicit the nuances of human reasoning.

Problem

• Investigates syllogistic reasoning in large language models
• Explores symbolic and natural language inference capabilities
• Questions if models become formal rather than human-like reasoners

Innovation

• Analyzing syllogistic reasoning in 14 LLMs
• Comparing symbolic inference and natural language understanding
• Investigating formal reasoning versus human reasoning nuances