Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy

2026-06-25

AI Summary

This work addresses the challenge of enabling embodied agents to operate persistently in unstructured environments by coordinating heterogeneous cyber-physical tools and managing failures over extended execution horizons. The authors propose a hierarchical asynchronous architecture that, for the first time, decouples planning, memory, and verification into cooperative asynchronous units. A multimodal semantic planner routes skills, while an event-boundary-driven memory compression mechanism ensures sublinear context growth. An asynchronous visual preemption engine establishes a semantic closed loop for physical execution. Evaluated across two robotic platforms and four IoT devices on 40 long-horizon tasks, the system achieves significantly higher end-to-end success rates and maintains near-constant computational overhead even beyond 100,000 interaction tokens, enabling medium-scale open-source models to match the performance of closed-source systems.

Abstract

Building persistent embodied agents in unstructured environments demands unified orchestration of heterogeneous tools spanning both cyber (APIs, IoT) and physical (manipulation, navigation) domains, coupled with autonomous recovery from physical failures that inevitably arise over extended operation. Existing systems treat these as separate problems: VLM-based planners lack a unified cyber-physical action space, agent frameworks accumulate unbounded context that degrades temporal coherence, and VLA policies execute open-loop without detecting their own failures. We argue that persistent autonomy requires not a monolithic model but a hierarchical asynchronous architecture with explicit separation of planning, memory, and verification. To this end, we present OmniAct, a framework integrating a multimodal semantic planner for skill routing across unified action spaces, an adaptive hierarchical memory with event-boundary-driven compression for sub-linear context growth, and an asynchronous visual preemption engine that closes the semantic loop during physical execution. Across 40 real-world long-horizon tasks on two robotic platforms coordinating four IoT devices, OmniAct achieves consistent improvements in end-to-end success across all complexity levels, maintains near-flat token consumption over under 100k+ accumulated interaction tokens, and elevates mid-scale open-weight models to proprietary-level performance.

Problem

• embodied agents
• physical autonomy
• cyber-physical integration
• failure recovery
• long-horizon tasks

Innovation

• hierarchical asynchronous architecture
• unified cyber-physical action space
• adaptive hierarchical memory
• asynchronous visual preemption
• persistent embodied autonomy