Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence

2026-08-12

AI Summary

This work addresses the challenge of enhancing the spatial reasoning capabilities of frozen vision-language models (VLMs) without parameter updates or reliance on external tools. The authors propose a runtime self-reflection framework that distills successful interactions in a verifiable environment into transferable textual experiences, which are then retrieved to guide decision-making during inference. A key innovation is the introduction of a Transfer Reliability Score (TRS) that dynamically optimizes experience reuse. The framework integrates verifier-guided reflection, semantic filtering, similarity-TRS joint ranking, and experience distillation to enable self-evolving spatial intelligence under read-only deployment. Evaluated across five spatial reasoning benchmarks and four base VLMs, the method achieves state-of-the-art macro-average performance, outperforming existing approaches on the majority of metrics across 20 evaluations.

Abstract

Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability of VLM agents, existing work has mainly followed two lines. One line uses post-training methods, such as supervised fine-tuning and reinforcement learning. Another line adopts an agentic paradigm in which the model calls external spatial tools, such as depth estimation and 3D reconstruction tools, to gather intermediate spatial evidence. We study a complementary and underexplored route: Can a frozen VLM agent improve its spatial reasoning through \textbf{parameter-update-free self-evolution}, without depending on external expert spatial tools at inference time? We present \textbf{Spatial Memory Agent (SMA)}, an \textbf{experience-grounded runtime framework} that converts verified spatial experience into reusable transferable lessons. In a verifiable spatial environment, SMA queries the frozen VLM, obtains a predicted answer and reward, and uses \textbf{verifier-guided reflection} to distill compact transferable lessons from spatial experience. SMA further assigns each lesson a \textbf{Transfer Reliability Score (TRS)}, which is initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. During \textbf{read-only deployment}, SMA retrieves lessons by semantic filter and similarity-TRS combined ranking, allowing the retrieved memory to guide frozen model inference. Across five representative spatial benchmarks and four base VLMs, SMA achieves the highest macro average in every base-model block and the best accuracy among the evaluated methods in most of the 20 evaluations, establishing a practical parameter-update-free path for spatial self-evolution across the evaluated frozen model scales and environments.

Problem

• spatial intelligence
• frozen VLM
• parameter-update-free
• self-evolution
• spatial reasoning

Innovation

• parameter-update-free
• experience-grounded
• verifier-guided reflection
• Transfer Reliability Score
• spatial memory