Invariant Reasoning Directions in Latent Trajectories of Language Models

2026-06-27

AI Summary

This work addresses the instability and ambiguity of reasoning trajectories in language model hidden states, which are highly sensitive to input rephrasing, model versions, and perturbations, lacking consistent and transferable reasoning directions. To tackle this, the authors propose TILR, a training-free intervention framework that, for the first time, reveals the existence of low-dimensional invariant subspaces within implicit reasoning trajectories. TILR extracts these subspaces by contrasting strong and weak reasoning paths and applies adaptive alignment gating to intervene in hidden states. Experiments demonstrate that TILR significantly improves answer consistency under rephrasing (by approximately 10%) and reduces trajectory variance by up to 50% across six reasoning benchmarks, all while preserving original reasoning accuracy—thereby enabling effective identification and manipulation of stable reasoning structures.

Abstract

Latent reasoning models perform multi-step inference directly in hidden-state space, yet the structure of these latent reasoning trajectories remains poorly understood. We show that contrastive refinement signals between stronger and weaker reasoning trajectories exhibit a highly concentrated low-rank structure, while unconstrained latent updates remain sensitive to paraphrases, checkpoint choice, and trajectory perturbations. These observations suggest that latent reasoning trajectories contain stable invariant directions mixed with unstable instance-specific variation. We introduce \textbf{Trajectory-Invariant Latent Refinement (TILR)}, a training-free intervention framework for identifying and manipulating stable reasoning directions in latent space. TILR first learns a low-rank invariant subspace from contrastive trajectory differences across inputs, then constrains latent interventions to this subspace while suppressing poorly aligned updates through an adaptive alignment gate. Across six reasoning benchmarks, we find that a small number of latent directions explain most variation between strong and weak reasoning trajectories. Interventions on these directions causally improve reasoning consistency and reduce trajectory instability under paraphrases and perturbations. TILR improves answer consistency under paraphrase by ~10% and reduces latent trajectory variance by up to $50\%$ while preserving reasoning accuracy. These results support a geometric view of latent reasoning in which transferable reasoning behavior emerges from stable low-dimensional structure within hidden-state trajectories.

Problem

• latent reasoning
• invariant directions
• trajectory instability
• language models
• hidden-state space

Innovation

• latent reasoning
• low-rank structure
• invariant directions
• trajectory refinement
• alignment gate