Coherence Under Commitment: Probing Generalization and Vacuous Memorization in LLM Logical Reasoning

2026-06-19

AI Summary

Large language models often preserve superficial consistency in logical reasoning by avoiding definitive judgments, leading to inflated evaluation scores despite a lack of genuine decisiveness. This work proposes a dual-query evaluation paradigm that jointly accounts for consistency and decisiveness: it introduces a commitment score to quantify judgment quality, designs a certainty elicitation protocol based on normalized log-probabilities of YES/NO responses, and formulates a True/False/Uncertain ternary decision framework to characterize the trade-off between the two desiderata. Experiments across four open-source models (1B–3B) reveal a sharp frontier between consistency and coverage—for instance, Qwen2.5-3B exhibits a contradiction rate of merely 0.025 but only 7.4% coverage, whereas TinyLlama-1.1B achieves 79.4% coverage yet contradicts in every case. Moreover, the proposed CUC metric demonstrates strong generalization on LogiQA-v2 (ρ = 0.97).

Abstract

Large language models (LLMs) deployed for logical reasoning in knowledge-intensive domains exhibit a subtle but critical failure: coherence can be vacuously achieved through systematic abstention. A model that withholds commitment to either entailment or refutation satisfies negation consistency while providing no utility. We introduce Coherence Under Commitment (CUC), a dual-query evaluation paradigm that jointly measures consistency and decisiveness. CUC contributes three innovations: (1) a commitment score $c(\varphi) = p(\varphi) + p(\lnot\varphi)$ quantifying probability mass allocated to decisive outcomes; (2) a \textbf{deterministic elicitation protocol} via normalized YES/NO log probabilities, eliminating sampling variance; and (3) a 3-way decision framework (True/False/Uncertain) operationalizing the coherence-commitment trade-off into metrics. Experiments on four open-weight LLMs (1B-3B) across 204 FOLIO examples expose a sharp frontier. Qwen2.5-3B achieves near-zero contradiction ($\mathbb{E}[v_{\mathrm{neg}}]{=}0.025$) but only $7.4\%$ coverage, while TinyLlama-1.1B reaches $79.4\%$ coverage with violations on every example. Coherence-only evaluation would rank the abstaining model first; CUC exposes this as vacuous, and the frontier generalizes to LogiQA~v2 ($ρ{=}0.97$). We argue that evaluation must report both coherence and non-vacuous commitment and release a toolkit for standardized assessment.

Problem

• logical reasoning
• coherence
• vacuous memorization
• commitment
• evaluation

Innovation

• Coherence Under Commitment
• deterministic elicitation protocol
• commitment score
• three-way decision framework
• logical reasoning evaluation