AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agents

2026-05-18

AI Summary

Existing vision-language model (VLM) agents for long-horizon tasks rely heavily on textual memory and external teacher models, limiting their capacity for spatial reasoning and depriving them of dense visual feedback. This work proposes AtlasVA, a novel framework that, for the first time, retains reusable experience at the visual level, establishing a teacher-free visual skill memory system. Memory is organized into a three-tier structure comprising spatial heatmaps, visual exemplars, and symbolic textual skills. Leveraging trajectory statistics and lightweight grid-based heuristics, the framework autonomously evolves affinity and hazard maps, which are then used to generate potential-shaping rewards for reinforcement learning. By unifying perception, memory, and optimization, AtlasVA significantly outperforms text-centric baselines and state-of-the-art VLM agents across diverse tasks—including Sokoban, FrozenLake, 3D navigation, and robotic manipulation—with particularly pronounced gains in spatially intensive scenarios.

Abstract

Vision-language model (VLM) agents increasingly rely on memory-augmented reinforcement learning to reuse experience across long-horizon tasks, yet most existing frameworks store memory as text and depend on proprietary teacher models to summarize or refine it. This design is poorly matched to spatial decision making: geometric priors are compressed into lossy language, and sparse interaction is often supervised through delayed textual feedback rather than dense visually grounded signals. We argue that reusable experience for VLM agents should remain visually grounded. Based on this insight, we propose \textbf{AtlasVA}, a teacher-free visual skill memory framework that organizes memory into three complementary layers: spatial heatmaps, visual exemplars, and symbolic text skills. AtlasVA further evolves danger and affinity atlases directly from trajectory statistics and lightweight grid heuristics, and reuses these self-evolving atlases as potential-based shaping rewards for reinforcement learning. This unifies perception, memory, and optimization without external LLM supervision. Experiments on \textsc{Sokoban}, \textsc{FrozenLake}, 3D embodied navigation, and 3D robotic manipulation benchmarks show that AtlasVA consistently outperforms text-centric memory baselines and competitive VLM agents, with especially strong gains on spatially intensive tasks. Homepage: https://wangpan-ustc.github.io/AtlasvaWeb

Problem

• visual grounding
• memory-augmented reinforcement learning
• spatial decision making
• teacher-free learning
• vision-language models

Innovation

• visual skill memory
• teacher-free VLM
• self-evolving atlas
• spatial heatmaps
• potential-based reward shaping