Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework

2026-05-23

AI Summary

Current evaluations of large language models overemphasize answer correctness, failing to capture the quality of their reasoning processes. This work proposes the first theory-driven, multidimensional framework for assessing reasoning quality, evaluating models across six behavioral dimensions: correctness, consistency, robustness, logical coherence, efficiency, and stability. Through large-scale experiments spanning 975 tasks and seven mainstream models, the framework demonstrates its effectiveness by revealing that logical coherence is orthogonal to correctness, leading to significant reversals in model rankings when accuracy alone is not the sole criterion. These findings provide fine-grained, actionable insights for model auditing and deployment decisions.

Abstract

LLMs have achieved remarkable success in complex reasoning tasks, yet current evaluation approaches predominantly rely on final-answer correctness, offering limited insight into the underlying reasoning processes that produce those answers. To address this gap, this study proposes a unified multi-dimensional framework for measuring reasoning quality in LLMs from a behavioral perspective, operationalizing six theoretically grounded dimensions: Correctness (CQ), Consistency (CS), Robustness (RS), Logical Coherence (LS), Efficiency (ES), and Stability (SS). Extensive experiments on seven LLMs across 975 items from four benchmarks demonstrate that the framework reveals behaviors invisible to accuracy-only metrics. Notably, logical coherence is orthogonal to correctness (r = -0.172, ns), confirming that correct answers can arise from incoherent reasoning, while Claude-Haiku-4.5 achieves the highest multi-dimensional score (Q_bal = 0.778). Furthermore, the framework exposes critical ranking inversions: DeepSeek-V3 ranks second under accuracy-priority but fifth under legal/compliance weighting, a reversal that single-metric evaluation cannot detect. Discriminant validity confirms 11/15 dimension pairs are independent (|r| < 0.50), providing psychometric support for treating each dimension as a distinct signal. The dimensional profiles produced by the framework directly support three classes of deployment decision: identifying models whose reasoning traces would fail accountability audits despite correct final answers (LS--CQ orthogonality); preventing ranking errors caused by accuracy-only benchmarking; and ensuring that no single metric silently substitutes for the six independent signals the framework captures.

Problem

• reasoning quality
• large language models
• evaluation metrics
• behavioral framework
• multi-dimensional assessment

Innovation

• multi-dimensional evaluation
• reasoning quality
• behavioral framework
• logical coherence
• discriminant validity