Inverting the Bellman Equation: From $Q$-Values to World Models

2026-06-19

AI Summary

Traditionally, model-free and model-based reinforcement learning have been regarded as mutually exclusive paradigms, with ambiguity surrounding whether value functions implicitly encode environmental dynamics. This work proposes P-learning, a novel approach that establishes, for the first time, an inverse mapping from goal-conditioned Q-functions to the environment’s transition kernel by inverting the Bellman equation. The method enables a unified analysis across both stochastic and deterministic MDPs, as well as discrete and continuous state spaces. By revealing that agents can implicitly learn a world model from sparse reward signals alone, P-learning bridges the conceptual divide between model-free and model-based RL. Empirical validation in Reacher, MountainCar, and FourRooms environments demonstrates its ability to achieve near-optimal generalization to out-of-distribution goals.

Abstract

Model-based and model-free reinforcement learning are traditionally viewed as separate paradigms: instead of learning a model of the transition kernel $P$, model-free agents typically estimate value functions tied to a specific policy and reward. In this paper, we challenge this dichotomy by proving that value-based agents trained on a sufficiently rich set of reward functions, e.g. using goal-conditioned RL, implicitly encode a unique and accurate world model. To extract this model in practice, we introduce \textit{$P$-learning}, an inverse analogue to $Q$-learning that samples from an agent's $Q$-values, policies and rewards to decode its internal model of the environment. We then provide sufficient conditions on the type and number of goals for which agents encode the true kernel $P$, covering both stochastic and deterministic MDPs over finite or continuous state spaces. Even when our assumptions are violated, we empirically demonstrate that agents trained on a handful of reward functions encode accurate dynamics in $\texttt{Reacher}$, $\texttt{MountainCar}$ and stochastic variants of $\texttt{FourRooms}$. Surprisingly, we find that policies trained exclusively on a \texttt{Reacher} agent's implicit world model are quasi-optimal on out-of-distribution, velocity-based goals despite position-only training -- suggesting that agents contain hidden generalisation capabilities and providing a new lens into the connection between model-based, model-free, and goal-conditioned RL.

Problem

• model-based reinforcement learning
• model-free reinforcement learning
• world model
• Bellman equation
• goal-conditioned RL

Innovation

• P-learning
• world model
• goal-conditioned RL
• inverse Bellman equation
• model-free to model-based