HoloAgent-0: A Unified Embodied Agent Framework with 3D Spatial Memory

2026-06-22

AI Summary

This work addresses the challenges of effectively scaling large language model agents to physical robots—namely, continuous action spaces, embodiment dependencies, environmental uncertainty, and safety constraints—by proposing HoloAgent-0, a unified embodied agent framework. Unlike existing systems that rely on isolated or loosely coupled architectures, HoloAgent-0 integrates operating system–level scheduling, 3D spatial memory, and heterogeneous robotic skills through an Embodied AgentOS that translates natural language instructions into executable skill graphs. The framework incorporates closed-loop feedback, real-time monitoring, and dynamic replanning mechanisms to ensure robust execution. Validated on real hardware, it successfully demonstrates long-horizon tasks including navigation, object search, mobile manipulation, and multi-robot collaboration, showcasing its capability in cross-task and cross-platform spatial memory modeling and closed-loop operation.

Abstract

LLM agents follow a practical execution loop in digital environments: they reason over structured states, invoke tools, inspect feedback, and revise actions. Extending this loop to physical robots is difficult because physical execution is continuous, embodiment-dependent, uncertain, and constrained by safety. Existing embodied-AI systems have advanced manipulation, spatial understanding, navigation, and humanoid control, but these capabilities often remain specialized modules or loosely coupled decision loops. In this work, we introduce HoloAgent-0, a unified embodied agent framework for real-world robot deployment. Embodied AgentOS converts language instructions into executable skill graphs, schedules robot resources, monitors execution, and triggers clarification or re-planning from runtime feedback. HoloAgent-0 organizes heterogeneous robot models and controllers through three coupled layers: Embodied AgentOS for closed-loop execution, 3D spatial memory for physical world grounding, and embodied skills for robot action. We deploy HoloAgent-0 on real hardware and evaluate its spatial memory, long-horizon navigation, and closed-loop execution across motion generation, object search, cross-robot coordination, and mobile manipulation.

Problem

• embodied AI
• 3D spatial memory
• robotic execution
• unified agent framework
• physical world grounding

Innovation

• Unified Embodied Agent
• 3D Spatial Memory
• Closed-loop Execution
• Skill Graph
• Embodied AgentOS