HiMe: Hierarchical Embodied Memory for Long-Horizon Vision-Language-Action Control

2026-07-03

AI Summary

This work addresses the limitations of existing vision-language-action (VLA) models in non-Markovian, long-horizon tasks, where reliance on immediate observations impedes long-term memory and reasoning, while high-capacity reasoning models struggle to meet real-time control demands—creating a “frequency–capability paradox.” To resolve this, the authors propose HiMe, a hierarchical embodied memory framework that decouples the system into a high-frequency executor, a working-memory sentinel, and a long-term planner. HiMe incorporates a dynamic knowledge system grounded in cross-modal semantic schemas, enabling memory plasticity through Add, Update, and Delete operations. This approach achieves the first explicit hierarchical decoupling of execution and planning, supports human-preference-driven self-correction, and significantly improves task success rates and adaptability in complex environments—all while maintaining real-time responsiveness and outperforming flat memory baselines.

Abstract

Current Vision-Language-Action (VLA) models excel at robotic manipulation but often struggle with non-Markovian tasks requiring long-term memory and reasoning due to their reliance on immediate observations. Existing solutions face a ''frequency-competence paradox,'' where stronger reasoning models are too slow for real-time control, while faster models lack sufficient reasoning capabilities. To resolve this architectural misalignment, we propose HiMe, a Hierarchical Embodied Memory framework that decouples embodied intelligence into a high-frequency Executor for execution, a Sentry for working memory, and a Planner for long-term strategy. We also introduce a dynamic knowledge system based on cross-modal semantic schemas and active management mechanisms, allowing robots to maintain memory plasticity through ''Add, Update, and Delete'' operations. This hierarchical design effectively balances the conflict between real-time execution and slow thinking planning, significantly improving success rates in long-horizon tasks. Experiments demonstrate that this approach not only outperforms flat memory baselines but also exhibits the novel ability to self-correct its internal knowledge based on human preferences.

Problem

• Vision-Language-Action
• long-horizon tasks
• non-Markovian tasks
• real-time control
• reasoning capabilities

Innovation

• Hierarchical Embodied Memory
• Vision-Language-Action
• Long-Horizon Planning
• Dynamic Knowledge System
• Memory Plasticity