EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies

2026-06-16

AI Summary

Current evaluations of general-purpose mobile manipulation policies rely predominantly on a single success rate metric, which fails to capture nuanced differences in multidimensional capabilities and generalization. To address this limitation, this work introduces EBench—a simulation-based multitask evaluation benchmark that establishes the first fine-grained diagnostic framework encompassing five capability dimensions and four generalization dimensions. Through capability annotation, distribution shift analysis, and cross-policy comparison, the framework reveals significant heterogeneity among high-success-rate models in their capability profiles and generalization behaviors: for instance, π₀.₅ exhibits the best generalization retention, InternVLA-A1 excels in locomotion but falters in dexterous manipulation, and XVLA demonstrates exceptional performance on specific atomic skills. These insights transcend the constraints of conventional aggregate scoring.

Abstract

We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies beyond a single success-rate scalar. EBench comprises 26 diverse and challenging manipulation tasks annotated along 5 capability dimensions and 4 generalization dimensions. We evaluate state-of-the-art generalist manipulation models including $π_0$, $π_{0.5}$, XVLA, and InternVLA-A1, and reveal that models with near success rates exhibit strikingly different capability profiles: $π_{0.5}$ achieves the highest test success rate and the best train--test retention, whereas InternVLA-A1 dominates mobile manipulation but collapses on dexterous tasks, and XVLA exhibits strengths on a disjoint set of atomic skills compared to other policies. Beyond capability profiling, EBench analyzes the generalization ability from 4 representative perspectives, identifying the impact of different distribution shift factors. The results reveal strengths and weaknesses of models behind an overall score. We hope this benchmark offers a broad set of diagnostic signals to guide iteration on generalist manipulation models.

Problem

• mobile manipulation
• generalist policies
• benchmarking
• capability diagnosis
• generalization

Innovation

• mobile manipulation
• generalist policy
• capability profiling
• generalization analysis
• simulation benchmark