Do Diagrams Help Large Language Models Reason? Evidence from Syllogistic Reasoning

2026-07-26

AI Summary

This study investigates whether diagrammatic representations enhance large language models’ performance in syllogistic reasoning. By systematically comparing four representational formats—natural language, logical symbols, linear diagrams, and Euler diagrams—across 285 syllogism problems, the authors evaluate the reasoning capabilities of Claude 3.5 Sonnet and GPT-4o-mini. This work presents the first systematic analysis of how diverse visual and symbolic representations influence logical reasoning in large models. Contrary to expectations, diagrams do not consistently improve performance: models excel on entailment and contradiction tasks but struggle significantly with neutral relations, frequently committing conversion errors. These findings reveal a systematic limitation in the models’ ability to handle indeterminate logical relationships, underscoring challenges in their reasoning over non-deterministic semantic structures.

Abstract

Diagrams are widely used to support logical reasoning, and prior studies suggest that representations such as Euler diagrams can improve human reasoning performance. Recent work has also explored their effects on large language models (LLMs). In this paper, we compare four representational conditions for syllogistic reasoning: natural language, logical notation, linear diagrams, and Euler diagrams. Using 285 problems from Ando et al. (2024), we evaluate two contemporary LLMs, Claude 3.5~Sonnet and GPT-4o-mini. Our results show that diagrammatic representations do not consistently improve performance. Although the models perform well on entailment and contradiction problems, they struggle with neutral problems and often make systematic conversion errors. Overall, the results suggest that the tested models gain limited benefit from diagrams in logical reasoning tasks.

Problem

• diagrams
• large language models
• syllogistic reasoning
• logical reasoning
• Euler diagrams

Innovation

• diagrammatic reasoning
• large language models
• syllogistic reasoning
• Euler diagrams
• logical inference