Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers

2026-06-19

AI Summary

This work addresses the gap between decodability and faithfulness in natural language reasoning, where generated rationales may not accurately reflect a model’s true reasoning process. The authors propose a verifier-coupled reasoning framework that interleaves inline assertions within reasoning trajectories and trains an auxiliary consistency head to predict the outputs of programmatic verifiers from hidden states of rationale segments. This approach leverages consistency training both as a representational shaping mechanism and a diagnostic tool. Empirical validation across multiple domains demonstrates its efficacy: LeanCheck achieves perfect directional disentanglement, KataGo commentaries attain 81% accuracy in win-rate bucket prediction, and code-related tasks exhibit a 98.6% coupling rate. Activation patching further reveals causal influences ranging from 73% to 89%, indicating that consistency loss particularly enhances fine-grained assertion fidelity.

Abstract

Language models can generate plausible rationales for their predictions, but these explanations may not faithfully represent the model's internal reasoning. We propose verifier-coupled reasoning, a framework that inserts inline claims into reasoning traces and trains an auxiliary consistency head to predict programmatic verifier outputs from rationale-span hidden states. The central finding is a gap between decodability and faithfulness: consistency training reliably makes verifier information decodable from rationale representations, but decodability does not guarantee faithful generation. In LeanCheck (formal theorem proving), rationale-only and proof-only pooling achieve perfect directional separation under counterfactual conflict. In KataGo (Go engine), commentary spans encode 10-way win-rate buckets at 81% accuracy. Yet in a code setting, the model achieves 98.6% coupling while its generated explanations remain unfaithful: fluent prose with correct structured claims, but describing unrelated algorithms; a controlled pretrained-vs-from-scratch comparison shows the gap is not capacity-driven. Synthetic activation patching confirms causal influence (73-89% vs. 31% baseline), FEVER reveals that evidence-only pooling isolates genuine evidence sensitivity at the cost of raw accuracy, and per-claim analysis shows that consistency loss disproportionately benefits fine-grained claims over binary ones. These results establish that consistency losses are effective diagnostics and representation-shaping tools, but not sufficient conditions for faithful reasoning.

Problem

• faithfulness
• natural-language rationales
• decodability
• programmatic verifiers
• reasoning traces

Innovation

• verifier-coupled reasoning
• faithfulness
• decodability
• consistency loss
• programmatic verifiers