The Value Axis: Language Models Encode Whether They're on the Right Track

2026-06-15

AI Summary

This study investigates whether large language models internally encode the value of current behavioral trajectories—defined as the likelihood that a policy achieves its goal—in a linearly representable manner. By constructing synthetic contextual reinforcement learning data and applying activation analysis alongside causal interventions, the authors identify an interpretable “value axis” within Qwen3-8B. This axis effectively discriminates between high- and low-confidence responses, correct versus incorrect code generations, and backtracking behaviors. The work further demonstrates that this value signal can be modulated through Direct Preference Optimization (DPO) and supervised fine-tuning, thereby increasing the internal value assigned to rewarded behaviors. Notably, the study reveals that politically sensitive queries are systematically assigned low value during post-training phases.

Abstract

We investigate whether language models internally track the value of their current trajectory, defined as the likelihood that their ongoing strategy will achieve their goals. Using synthetic, in-context reinforcement learning data, we construct a "value" axis for Qwen3-8B. We find that activations along this axis distinguish between high vs. low verbalized confidence, rollouts without and with backtracking, and correct vs. corrupted code. Steering towards high value causally suppresses self-correction and reduces explanatory verbosity, while steering towards low value induces backtracking and exploration. We demonstrate that direct preference optimization (DPO) can increase the internal value of rewarded behaviors (e.g. use a certain word), causing the model to act more confidently after exhibiting them. Finally, we apply the value axis to study in-the-wild settings. For example, we find that Qwen assigns low value to politically sensitive chat queries after post-training and that supervised fine-tuning increases internal confidence within the training domain. Our results suggest that language models linearly encode an estimate of expected goal success that modulates their confidence in pursuing a direction.

Problem

• value estimation
• language models
• goal-directed behavior
• internal confidence
• trajectory evaluation

Innovation

• value axis
• internal confidence
• steering
• direct preference optimization
• in-context reinforcement learning