REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation

2025-12-22

AI Summary

Evaluating and improving the cross-environment generalization of Vision-Language-Action (VLA) models for robotic manipulation remains challenging due to the lack of realistic, real-to-sim-validated benchmarks. Method: We introduce REALM, a high-fidelity simulation benchmark featuring 15 environmental perturbations, 7 manipulation skills, and 3,500+ real-world-aligned objects, enabled by physics-based simulation, cross-domain-consistent control modeling, and a structured task-generation framework. We propose a standardized evaluation protocol covering mainstream VLA models—including π₀, π₀-FAST, and GR00T N1.5. Contribution/Results: Experiments demonstrate strong correlation (r > 0.92) between simulated performance and real-world behavior, exposing critical robustness deficits in current VLA models. REALM is the first open-source, reproducible, and extensible benchmark dedicated to VLA generalization evaluation, enabling rigorous, scalable assessment of cross-environment transfer capabilities.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Large Vision Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Vision-Language-Action (VLA) models empower robots to understand and execute tasks described by natural language instructions. However, a key challenge lies in their ability to generalize beyond the specific environments and conditions they were trained on, which is presently difficult and expensive to evaluate in the real-world. To address this gap, we present REALM, a new simulation environment and benchmark designed to evaluate the generalization capabilities of VLA models, with a specific emphasis on establishing a strong correlation between simulated and real-world performance through high-fidelity visuals and aligned robot control. Our environment offers a suite of 15 perturbation factors, 7 manipulation skills, and more than 3,500 objects. Finally, we establish two task sets that form our benchmark and evaluate the π_{0}, π_{0}-FAST, and GR00T N1.5 VLA models, showing that generalization and robustness remain an open challenge. More broadly, we also show that simulation gives us a valuable proxy for the real-world and allows us to systematically probe for and quantify the weaknesses and failure modes of VLAs. Project page: https://martin-sedlacek.com/realm

Problem

• Evaluates generalization of Vision-Language-Action models in robotics
• Addresses lack of real-world validated simulation benchmarks
• Systematically probes weaknesses in robotic manipulation skills

Innovation

• Simulation environment with high-fidelity visuals and control
• Benchmark with perturbation factors and manipulation skills
• Validated real-to-sim correlation for generalization evaluation