Agent as Policy for Robotic Manipulation

2026-09-11

AI Summary

本文提出了一种名为Agent as Policy (AGP)的方法，通过让通用智能体直接控制物理机器人完成任务，无需特定任务或环境的额外训练，解决了机器人操作中的泛化问题。

Abstract

We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. AGP achieves success rates of 100%, 100%, and 80% on three block construction configurations. These findings establish a path for general-purpose agents to act as robotic policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.

Problem

• General-purpose agent
• Robotic manipulation
• Task execution
• Physical world interaction
• Runtime reasoning

Innovation

• Agent as Policy
• runtime reasoning
• programming
• continuous interaction