Problem
Research questions and friction points this paper is trying to address.
language models
on-policy recognition
output entropy
post-training
self-generation
From Simulation to Enaction: Post-trained language models recognize and react to their own generations
🤖 AI Summary
This work addresses the limitation of language models in pretraining to model the consequences of their own generations and to discern whether they operate within or outside their policy distribution. Through post-training interventions, the study demonstrates for the first time that models can implicitly distinguish between in-policy and out-of-policy contexts based on their own outputs, revealing a dissociation between explicit and implicit recognition mechanisms. The authors propose a novel mechanism that dynamically modulates output entropy according to the “surprisal” of the input representation: during in-policy generation, output entropy decreases by a factor of 3–4, and topic intent converges even before the first generated token; deviations from this intent lead to a marked entropy increase. This phenomenon is validated across multiple model families and scales, confirming its generality.
📝 Abstract
Language models are pretrained as passive predictors with no incentive to model the consequences of their own outputs. Post-training changes this: a model producing its own responses can benefit from recognizing that it is on-policy. We present evidence that post-trained models recognize their on-policy generations, and this recognition is implicitly encoded in their output distributions. In particular, on-policy output distribution entropy is 3--4$\times$ lower than off-policy entropy, across model families and size classes. We trace part of this effect to an internal representation of input surprise, tracking the unlikeliness of the most recent input token according to the model's prior predictions, that causally modulates output entropy. One example of these phenomena can be observed in response to open-ended prompts; post-trained models (unlike pretrained models) collapse their uncertainty over the topic of their upcoming response before the first output token; violating this cached intention with a different-topic prefill results in higher output entropy. We also tested whether models can distinguish on-policy contexts from prefills via explicit verbal report. We find that they can, but that interestingly, this explicit recognition routes through a different mechanism than implicit recognition.
Innovation
Methods, ideas, or system contributions that make the work stand out.
post-training
on-policy recognition
output entropy
input surprise
language models
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
