Mind Dreamer: Untethering Imagination via Active Latent Intervention on Latent Manifolds

2026-05-15

AI Summary

This work addresses the “history entanglement” problem in model-based reinforcement learning, where policy optimization lags behind the world model’s exploration of the latent manifold due to historical state initialization. To overcome this, we propose Active Latent Intervention (ALI), which relaxes the Markov continuity assumption by sampling non-consecutive yet physically plausible latent jumps via a generator to construct cognitive challenges. ALI introduces a Relay Value Function (RVF) and a Relay Uncertainty Function (RUF) to enable credit assignment across discontinuities. We formalize that uncertainty propagation in non-continuous imagination requires a quadratic discount factor γ² and establish a relay manifold expected free energy minimization framework, effectively widening the spectral gap of the latent manifold and reducing hitting time to critical bottleneck states. Integrated into DreamerV3 with an adversarial generator and Bellman-style propagation, ALI achieves 1.67× average speedup on the DeepMind Control Suite and up to 8.8× in sparse-reward tasks, significantly enhancing sample efficiency and exploration.

Abstract

Model-Based Reinforcement Learning (MBRL) leverages latent imagination for sample efficiency, yet remains constrained by Historical Tethering: imagination is typically initialized from observed states. This creates a learning asymmetry, where the world model's manifold discovery outpaces the policy's sparse-reward optimization. We propose Mind Dreamer (MD), a framework that operationalizes Active Latent Intervention (ALI) to transcend Markovian continuity. MD reformulates discovery as the minimization of a global Relay Manifold Expected Free Energy (R-EFE); by sampling initial states from a learned generator $s_0 \sim p_{gen}(\cdot)$ rather than the historical buffer, MD utilizes an adversarial generator to synthesize non-continuous latent jumps to epistemic blind spots that are physically plausible yet cognitively challenging. To resolve the credit assignment paradox across these spatial ruptures, we derive the Relay Value Function (RVF) and Relay Uncertainty Function (RUF). These potentials treat synthesized anchors as counterfactual intermediary states, propagating pragmatic and epistemic value through a principled Bellman-style formulation. Notably, we prove that uncertainty propagation across discontinuities necessitates a quadratic discount $\gamma^2$, establishing a formal epistemic horizon. Theoretically, MD approximates a variance-minimizing importance sampler that expands the manifold's spectral gap, reducing the hitting time to critical bottleneck states. Empirically, MD achieves a 1.67$\times$ average speedup over DreamerV3 on DeepMind Control Suite, reaching 8.8$\times$ in sparse-reward tasks.

Problem

• Historical Tethering
• Model-Based Reinforcement Learning
• Latent Manifolds
• Sparse-Reward Optimization
• Imagination

Innovation

• Active Latent Intervention
• Relay Manifold Expected Free Energy
• Relay Value Function
• Latent Manifold Exploration
• Model-Based Reinforcement Learning