Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference

2026-07-09

AI Summary

To address the limitations of language-guided continuous control in spatial reasoning and fine-grained motor execution, this work proposes the Latent Memory Palace (LMP) framework, which formulates policy reasoning as an autoregressive variational inference problem, iteratively organizing and retrieving information in a latent space. LMP is the first approach to integrate autoregressive latent variables with variational inference to enable interpretable, adaptive computation allocation at test time. It further introduces LMP-tok, a variable-length action tokenizer that enhances policy expressiveness. Experimental results demonstrate that the proposed policy, LMP-π, significantly outperforms baseline methods in both simulated and real-world tasks, validating its effectiveness in flexible reasoning and control performance.

Abstract

Human decision-making is highly flexible -- some actions are taken immediately; others require longer deliberation. Language models have exhibited a similar capacity for adaptive "reasoning." However, transferring this capability to continuous control policies has been challenging, as directly reasoning in language space may lack the granularity for spatial understanding and precise motions. In this work, we show that reasoning for control policies can emerge by organizing information in an autoregressive latent space reminiscent of a memory palace, where retrieval is iterative and adaptive. Our method, Latent Memory Palace (LMP), formulates reasoning as variational inference with an autoregressive latent distribution. We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound. The resulting policy, LMP-$π$, achieves strong empirical performance in simulation and real-world domains while exhibiting interpretable, adaptive allocation of test-time compute. We further show that the same framework yields a variable-length action tokenizer, LMP-$\texttt{tok}$, which significantly improves the performance of downstream autoregressive policies. Together, these results present a new perspective on latent reasoning for control through the lens of variational inference.

Problem

• reasoning
• continuous control
• latent space
• autoregressive
• variational inference

Innovation

• Latent Memory Palace
• autoregressive variational inference
• reasoning for control
• latent-space reinforcement learning
• variable-length action tokenization