RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

2026-06-22

AI Summary

This work addresses the challenge that videos generated by existing robotic world models often violate physical laws, temporal consistency, or task logic, while conventional metrics and general-purpose vision-language models (VLMs) struggle to diagnose such errors accurately. To overcome this, we propose RoboGaze—a training-free, multi-agent vision-language framework that enables structured evaluation through a three-stage pipeline: task-scenario alignment, dimension-specific expert routing, and critique-based validation. We introduce the first six-dimensional, thirty-category error taxonomy tailored for robotics, facilitating fine-grained and interpretable detection of temporal-local anomalies and substantially reducing false positives from generic VLMs. Evaluated on a dataset of 382 human-annotated videos, our approach improves description F1 by 43 points, boosts temporal alignment metrics by 37 points, and elevates accuracy on clean videos from below 25% to over 80%, reaching 85% of human evaluation performance.

Abstract

Recent advances in robot world models enable synthetic video generation for embodied prediction and planning. However, evaluating these videos is challenging: visually realistic outputs often violate physical laws, temporal consistency, or task logic, while conventional metrics and monolithic Vision-Language Model (VLM) judges fail to generalize or provide precise diagnostic value. We present RoboGaze, a training-free, multi-agent VLM framework that provides structured, interpretable evaluation for generated robot-manipulation videos. Given a task instruction and video, RoboGaze operates via a three-stage pipeline: task-scene grounding, dimension-specific specialist routing, and critic-based verification. It outputs temporally localized glitch reports categorized under a novel 6-dimension, 30-type robotics-specific taxonomy. To benchmark RoboGaze, we introduce a human-validated dataset of 382 clips spanning simulated and real-world multi-view manipulation. Evaluating eight open-source and proprietary VLM backbones, RoboGaze dramatically outperforms zero-shot baselines, improving description-F1 by up to +43 points and temporal alignment (F1 x IoU) by up to +37 points, closing approximately 85% of the gap to the human ceiling. Furthermore, its critic verifier mitigates the "cry-wolf" false-positive flaw of standard VLMs, lifting clean-clip accuracy from under 25% to over 80%. RoboGaze offers a scalable, highly interpretable diagnostic tool for the rigorous evaluation of robot world models.

Problem

• robot world models
• video evaluation
• physical consistency
• temporal coherence
• vision-language models

Innovation

• structured evaluation
• multi-agent VLM
• robot world models
• temporal glitch detection
• critic-based verification