Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty

2026-06-15

AI Summary

This work addresses the tendency of large language models to produce inconsistent or contradictory reasoning paths in multi-step logical tasks. While existing approaches assess reliability primarily through answer dispersion, they overlook the consistency signals inherent in the model’s own rankings of its generated reasoning paths. The authors propose a structural uncertainty framework that elicits multiple self-generated reasoning paths and pairwise preference judgments from the model, then aggregates these into a ranking distribution using the Bradley–Terry model and PageRank algorithm. This distribution is decomposed into two entropy-based metrics: cross-iteration instability and intra-iteration candidate ambiguity. Evaluated across five large models and eight benchmarks, the method demonstrates that structural consistency signals complement answer dispersion, significantly improving detection of unreliable instances in logical and mathematical reasoning. Notably, the signal becomes uniform in factual retrieval tasks, highlighting its specific sensitivity to reasoning consistency.

Abstract

Large language models can arrive at the same answer through reasoning paths that are unstable, contradictory, or difficult to rank consistently -- a failure mode especially prevalent in multi-step deductive reasoning. Existing methods assess reliability primarily through output dispersion -- measuring how much sampled answers differ -- but this discards a complementary signal: whether the model can consistently rank competing reasoning candidates. We propose structural uncertainty, a consistency-aware framework derived from the stability of self-preference-induced rankings over sampled reasoning solutions. Given a query, we generate multiple candidate solutions and ask the model to judge pairwise preferences among its own outputs. We aggregate self-preferences into ranking distributions via Bradley-Terry modeling with PageRank, and decompose the signal into two entropy-based components: across-trial ranking instability and within-trial candidate ambiguity. Across five LLMs and eight benchmarks, structural signals provide information complementary to answer dispersion: on logical and mathematical reasoning tasks, the combination improves identification of unreliable instances, while on factual retrieval the structural signal collapses toward uniformity, diagnosing a regime boundary where reasoning-level consistency evaluation is uninformative. The two components relate differently to accuracy: within-trial ambiguity correlates positively with correctness -- consistent with settings where multiple plausible solution paths remain competitive -- while across-trial instability correlates negatively, signaling unreliable reasoning. Structural uncertainty is best understood not as a universal confidence estimator, but as a regime-sensitive evaluator of logical reasoning consistency.

Problem

• logical reasoning
• consistency
• large language models
• reasoning paths
• structural uncertainty

Innovation

• structural uncertainty
• logical reasoning consistency
• self-preference ranking
• Bradley-Terry modeling
• entropy-based decomposition