ACE: Agentic Control for Embodied Manipulation via Zero-shot Workflow Reasoning

2026-07-05

AI Summary

This work addresses the challenges of language grounding, environmental dynamics, and execution recovery in open-world tabletop manipulation tasks by proposing a zero-shot workflow reasoning framework. The approach integrates semantic subgoal generation, a mask-mediated visuomotor interface, and a closed-loop multi-temporal memory mechanism to jointly enable explicit workflow reasoning and mask-driven executable skills—without requiring task-specific fine-tuning. Leveraging a visually grounded interface, reusable grasp/place primitives, and a task-agnostic downstream policy, the system supports real-time human-in-the-loop corrections. Evaluated on complex logical tasks, it substantially outperforms end-to-end baselines, achieving a 50% success rate on equation construction and 70% on constrained retrieval tasks.

Abstract

Open-ended tabletop manipulation requires agents to not only understand natural language but also adapt to dynamic environments and execution failures. We present ACE (Agentic Control for Embodied Manipulation), a zero-shot workflow reasoning framework for tabletop pick-and-place from natural language. Rather than relying on direct low-level action mapping, ACE combines agentic workflow reasoning with two robot-facing executable skills: a visual grounding interface and a reusable pick-and-place primitive. To bridge semantic reasoning and physical control, the active sub-goal is grounded into a mask-mediated vision-action interface. This unified mask specifies the target object and destination, is tracked over time, exposed for human verification, and ultimately passed to a task-agnostic downstream policy for execution. Crucially, ACE operates in a closed loop supported by a multi-timescale memory. After an action is executed, the system automatically verifies whether the intended sub-goal succeeded, using the outcome to advance, retry, repair, or replan. This enables online adaptation to user corrections, scene changes, and physical failures. We evaluate ACE on logically complex, long-horizon tasks, including zero-shot multi-step equation formation with number cubes and constraint-based object retrieval. ACE demonstrates task-level zero-shot generalization on novel semantic constraints and randomized tabletop scenes without task-specific retraining. Specifically, while standard end-to-end baselines struggle to complete these logically demanding tasks, ACE achieves a 50% success rate in equation formation and a 70% success rate in constraint retrieval. This contrast demonstrates that explicit workflow reasoning and mask-mediated control offer a robust, practical route toward adaptable robotic manipulation.

Problem

• embodied manipulation
• zero-shot reasoning
• natural language understanding
• dynamic environment adaptation
• execution failure recovery

Innovation

• zero-shot workflow reasoning
• mask-mediated control
• agentic manipulation
• closed-loop adaptation
• visual grounding