A Taxonomy for Evaluating Generalist Robot Policies

2025-03-03

AI Summary

Existing evaluations of robot policy generalization lack standardized, reproducible benchmarks—particularly for manipulation tasks, where visual, semantic, and behavioral generalization remain difficult to quantify. To address this, we propose STAR-Gen: the first fine-grained, structured, and empirically grounded taxonomy for robot policy generalization. Leveraging Bridge V2, we construct the first standardized benchmark explicitly designed for real-world robotic manipulation. Through systematic evaluation of multiple state-of-the-art Vision-Language-Action (VLA) models on physical robots, we identify a critical bottleneck in semantic generalization. We publicly release comprehensive evaluation protocols, annotated demonstration videos, and interactive execution logs. This work provides both a theoretical framework and practical tools to advance robot generalization research, enabling rigorous, comparable, and reproducible assessment across methods. (138 words)

Technology Category

• Intelligent Robots: Manipulation
• Natural Language Processing: Generation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Machine learning for robotics promises to unlock generalization to novel tasks and environments. Guided by this promise, many recent works have focused on scaling up robot data collection and developing larger, more expressive policies to achieve this. But how do we measure progress towards this goal of policy generalization in practice? Evaluating and quantifying generalization is the Wild West of modern robotics, with each work proposing and measuring different types of generalization in their own, often difficult to reproduce, settings. In this work, our goal is (1) to outline the forms of generalization we believe are important in robot manipulation in a comprehensive and fine-grained manner, and (2) to provide reproducible guidelines for measuring these notions of generalization. We first propose STAR-Gen, a taxonomy of generalization for robot manipulation structured around visual, semantic, and behavioral generalization. We discuss how our taxonomy encompasses most prior notions of generalization in robotics. Next, we instantiate STAR-Gen with a concrete real-world benchmark based on the widely-used Bridge V2 dataset. We evaluate a variety of state-of-the-art models on this benchmark to demonstrate the utility of our taxonomy in practice. Our taxonomy of generalization can yield many interesting insights into existing models: for example, we observe that current vision-language-action models struggle with various types of semantic generalization, despite the promise of pre-training on internet-scale language datasets. We believe STAR-Gen and our guidelines can improve the dissemination and evaluation of progress towards generalization in robotics, which we hope will guide model design and future data collection efforts. We provide videos and demos at our website stargen-taxonomy.github.io.

Problem

• Measure progress in robot policy generalization.
• Propose a taxonomy for evaluating robot manipulation generalization.
• Provide reproducible guidelines for measuring generalization in robotics.

Innovation

• Proposes STAR-Gen taxonomy for robot generalization
• Uses Bridge V2 dataset for real-world benchmarking
• Evaluates vision-language-action models on semantic generalization