Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation

2026-07-06

AI Summary

Existing vision-language-action (VLA) models struggle with long-horizon manipulation tasks due to their reliance solely on current observations, while hierarchical approaches often suffer from performance degradation caused by misalignment between high-level semantics and low-level kinematics. This work proposes Cortex, a bidirectionally aligned embodied agent framework that enables precise coordination between executable subtask plans generated by a high-level vision-language model (VLM) and a low-level VLA policy. Key innovations include a set of 32 standardized skill primitives, automatic annotation of over 4,000 hours of video data based on executability principles to generate simulation data, event-balanced sampling, and a context-to-skill-constrained inference mechanism. Cortex outperforms monolithic baselines by 3.1% on Libero-long and 4.1% on RoboTwin, and demonstrates zero-shot generalization to unseen long-horizon tasks such as multi-stage chemical experiments.

Abstract

While recent Vision-Language-Action (VLA) models show promise toward generalist manipulation policies, they struggle with long-horizon tasks due to their Markovian nature-relying solely on current observations. Hierarchical dual-system methods address this but suffer from a gap between high-level planning semantics and low-level execution kinematics. We introduce Cortex, a bidirectionally aligned embodied agent framework with a customized planning interface that conveys executable and tractable subtask plans from high-level VLM to low-level VLA. Specifically, we standardize manipulation subtasks into 32 canonical skill primitives and inject tractability principles, such as representative object attributes and improved trajectory reachability, into the data generation pipeline. This enables automatic annotation of over 4k hours of open-source video data and generation of 30 hours of simulation data. We further devise an event-balanced sampling strategy to construct training data for fine-tuning the framework to better handle planning ambiguity during subtask transitions, enhanced by carefully designed harness engineering from task contexts to skill constraints during inference. Both open-loop VLM and closed-loop system evaluations demonstrate Cortex's efficacy, e.g., it outperforms monolithic baselines by 3.1% on Libero-long and 4.1% on RoboTwin. Notably, Cortex's generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks, such as multi-stage chemistry experiments, by simply combining with a fine-tuned VLA-a capability infeasible through VLA fine-tuning alone.

Problem

• long-horizon manipulation
• Vision-Language-Action models
• hierarchical planning
• embodied agents
• skill primitives

Innovation

• bidirectional alignment
• skill primitives
• long-horizon manipulation
• Vision-Language-Action (VLA)
• zero-shot generalization