Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectories

2026-07-07

AI Summary

This study investigates how the causal faithfulness of implicit reasoning—defined as the extent to which intermediate reasoning steps causally influence the final answer—evolves throughout model training. By applying intervention techniques such as counterfactual input editing, activation ablation, and patching across multiple training checkpoints, the work systematically evaluates the causal contribution of reasoning steps. The analysis reveals, for the first time, that faithfulness is highly dependent on both training stage and answer format: while different tasks may exhibit similar surface-level unfaithful behaviors at output, their internal activation trajectories diverge substantially. Moreover, the causal influence of reasoning steps generally diminishes with training in binary-choice tasks but unexpectedly increases in open-ended generation settings.

Abstract

Latent reasoning methods perform multi-step inference entirely in the model's continuous hidden states, promising more compact and efficient reasoning. However, these opaque hidden states raise a question of faithfulness: whether these latent reasoning steps causally drive the final answer. Prior work investigates this question at converged checkpoints and reports several unfaithful behaviors, such as latent reasoning steps that can be replaced without changing the answer, but leaves how these behaviors form during training unexamined. We instead track how faithfulness evolves across saved checkpoints for different latent reasoning paradigms, applying a verifiable counterfactual edit on the input and a noise-ablation activation patch on the latent reasoning steps. We find that (i) at the output level, latent reasoning methods can look similarly unfaithful at convergence under counterfactual edits while following qualitatively divergent trajectories; (ii) at the activation level, the causal contribution of latent reasoning steps to the final answer decays across training for both paradigms, with the examples that flip on the output side in (i) also being the examples on which this contribution decays; and (iii) the activation-level trajectory diverges by answer format, decaying on binary choice and rising on open-ended decoding. These findings highlight that latent reasoning faithfulness depends on training stage and answer format.

Problem

• latent reasoning
• faithfulness
• training trajectory
• causal contribution
• answer format

Innovation

• latent reasoning
• faithfulness
• training trajectory
• counterfactual editing
• activation ablation