🤖 AI Summary
This study investigates how world models in reinforcement learning acquire structured representations of environmental states. Focusing on the IRIS and DIAMOND architectures trained on Atari games, the work presents the first systematic evaluation of their latent representations through a combination of linear and nonlinear probing, causal interventions, attention head analysis, and multi-baseline token ablation experiments. The findings reveal that both models learn approximately linear encodings of key state variables—such as object positions and game scores—and that these representations are effectively leveraged by downstream policies. Furthermore, IRIS exhibits spatially specific attention heads, indicating a mechanism that selectively focuses on localized regions of the environment, thereby offering insight into how such models attend to and process structured visual information.
📝 Abstract
World models learn to simulate environment dynamics from experience, enabling sample-efficient reinforcement learning. But what do these models actually represent internally? We apply interpretability techniques--including linear and nonlinear probing, causal interventions, and attention analysis--to two architecturally distinct world models: IRIS (discrete token transformer) and DIAMOND (continuous diffusion UNet), trained on Atari Breakout and Pong. Using linear probes, we find that both models develop linearly decodable representations of game state variables (object positions, scores), with MLP probes yielding only marginally higher R^2, confirming that these representations are approximately linear. Causal interventions--shifting hidden states along probe-derived directions--produce correlated changes in model predictions, providing evidence that representations are functionally used rather than merely correlated. Analysis of IRIS attention heads reveals spatial specialization: specific heads attend preferentially to tokens overlapping with game objects. Multi-baseline token ablation experiments consistently identify object-containing tokens as disproportionately important. Our findings provide interpretability evidence that learned world models develop structured, approximately linear internal representations of environment state across two games and two architectures.