Colosseum V2: Benchmarking Generalization for Vision Language Action Models

2026-05-26

AI Summary

This work addresses the limited generalization of current vision-language-action (VLA) models under distribution shifts and the absence of a systematic evaluation benchmark. To this end, we introduce Colosseum V2, a large-scale simulated benchmark encompassing 28 tasks, 13 manipulation primitives, and two robot morphologies, which for the first time enables standardized evaluation of both in-distribution and out-of-distribution generalization within a unified framework. Built upon the ManiSkill simulator and accelerated with GPU rendering, the benchmark integrates state-of-the-art methods such as ACT and Pi0.5, demonstrating strong sim-to-real correlation. Our experiments reveal fundamental limitations of existing VLA models in both basic performance and generalization, establishing Colosseum V2 as a new platform for fair, efficient, and reproducible algorithmic comparison.

Abstract

Vision-Language-Action (VLA) models demonstrate promising generalization in robotic manipulation, driven by advances in large-scale vision and language pre-training. This progress can be misleading. Despite the zero-shot perception and language capabilities of VLAs, their overall task performance often degrades under distribution shifts, revealing gaps in how these systems translate high-level understanding into robust behavior. To systematically study this gap, we introduce Colosseum V2, a large-scale simulation benchmark for evaluating VLA generalization in robot learning across diverse conditions. The benchmark comprises 28 tasks spanning 13 task categories and two robot morphologies, covering a wide range of manipulation primitives and long-horizon behaviors. Built on the ManiSkill simulator, Colosseum V2 enables fast, GPU-parallelized evaluation and supports both in-domain and out-of-domain testing at scale. We evaluate state-of-the-art methods, including Action Chunking Transformers (ACT) and Pi0.5, and reveal limitations in both base performance and generalization. We demonstrate strong correlations between simulation and real-world metrics that support the ecological validity of the benchmark. By standardizing tasks, metrics, and evaluation protocols within a unified benchmark, Colosseum V2 enables reproducible and fair comparisons, reduced evaluation overhead, and accelerated progress toward general-purpose robot policies.

Problem

• Vision-Language-Action models
• generalization
• distribution shift
• robotic manipulation
• benchmark

Innovation

• Vision-Language-Action models
• generalization benchmark
• robotic manipulation
• simulation-to-reality transfer
• GPU-parallelized evaluation