WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation

2026-06-24

AI Summary

Existing robotic manipulation benchmarks struggle to evaluate a system’s ability to understand action sequences, intentions, and environmental dynamics from human demonstration videos. This work proposes the first benchmark specifically designed for behavioral understanding in robotic manipulation, integrating real human operation videos, natural language instructions, simulated environments, and executable tasks. It defines four cognitive dimensions—planning reasoning, policy execution, task completion, and contextual awareness—and introduces a decoupled evaluation protocol. Methodologically, the framework aligns video observations with simulation through the LIBERO task suite and leverages vision-language models such as Gemini-3.1-Pro alongside a robotic policy (π₀.₅). Experiments reveal that even state-of-the-art systems achieve only 16.3% and 14.0% task success rates in simulation and the real world, respectively, far below the human planning accuracy of 97.1%, underscoring the benchmark’s challenge and necessity.

Abstract

A robot working alongside people must reason about what they have done, in what order, and with what intent. Video carries the spatial layouts, object histories, and gestures that language leaves underspecified, yet today's manipulation benchmarks pair an instruction with a single current image, offering no way to evaluate reasoning over observed human behavior. We introduce WatchAct, a benchmark for robot manipulation grounded in observed human behavior. Each instance pairs a real-world human-action video and a language instruction with an aligned simulator scene and an executable LIBERO task, enabling scalable and reproducible evaluation. WatchAct comprises 3,000 long-horizon instances across 14 tasks in four capability domains drawn from the cognitive demands of watching another agent: parsing events (Event Grounding), recovering procedural structure (Procedural Reasoning), inferring unstated intent (Implicit Intent Inference), and tracking how the scene was changed (Episodic Reasoning). We further propose a disentangled evaluation protocol that separately measures (i)~video-to-plan reasoning by vision-language models, (ii)~policy execution under oracle plans, and (iii)~full task completion by integrated planner--policy pipelines. In both simulation and on a Franka Research 3 robot, current systems remain far from solving WatchAct. The best pipeline, Gemini-3.1-Pro with $π_{0.5}$, reaches only 16.3% Success Rate (SR) in simulation and 14.0% on the real robot. Gemini-3.1-Pro attains just 36.8% Plan SR (vs. 97.1% for humans), while $π_{0.5}$ reaches only 21.5% Task SR under oracle plans and drops to 10.6% on out-of-domain scenarios. Dataset and code are available at https://baiqi-li.github.io/watchact_page/.

Problem

• robot manipulation
• human behavior understanding
• video grounding
• intent inference
• procedural reasoning

Innovation

• behavior-grounded manipulation
• video-to-plan reasoning
• procedural reasoning
• implicit intent inference
• disentangled evaluation