StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models

2026-08-12

AI Summary

This work addresses the limited out-of-distribution (OOD) generalization of existing vision-language-action (VLA) models, which typically require extensive fine-tuning with new data. The authors propose StellaVLA, a framework that leverages a single structured demonstration retrieved at test time to automatically generate task plans, subgoal descriptions, and 3D action language, thereby guiding the policy toward task-level reasoning rather than pixel-level imitation. StellaVLA introduces the first annotation-free mechanism for generating structured demonstrations and employs a parallel dual-training architecture, enabling efficient OOD generalization across embodied platforms—including robots, human hands, and XR systems—without inference latency. Evaluated on VLA-Arena, StellaVLA achieves a top score of 0.63; it attains success rates of 98.8% on LIBERO and 85.1% on LIBERO-Plus, and demonstrates effective multi-source demonstration-driven OOD adaptation on real robots.

Abstract

Vision-Language-Action (VLA) models can follow instructions and manipulate objects, but their performance often collapses out of distribution (OOD), when the scene, viewpoint, or object differs from training. Adapting to each new situation typically requires collecting more data and fine-tuning. We present StellaVLA, a framework that instead adapts at test time by conditioning on a single retrieved demonstration. The key idea is to move beyond imitating what an expert did and instead convey why: an automated offline pipeline converts each raw trajectory into a structured demonstration, e.g., a task plan, sub-goal descriptions, and verbalized 3D motion, at zero human-annotation cost. Provided as in-context guidance, this structured demonstration lets the policy reason about the task rather than mimic a pixel trajectory, which also makes it transferable across embodiments (real-robot, human-hand, or XR demonstrations). A parallel dual-training design internalizes this reasoning during training through a joint action-and-language objective, while inference uses the action expert alone, preserving real-time, high-frequency control with no added latency. On the VLA-Arena leaderboard(Aug 1, 2026), StellaVLA ranks first with an overall score of 0.63, versus 0.44 and 0.22 for the strong prior models ($π_{0.5}$ and LingBot-VLA), and it further leads on LIBERO with 98.8% average success rate and LIBERO-Plus with 85.1% success rate. Our real-robot benchmark demonstrates that StellaVLA can use both human/robot demos and human-to-robot (XR) demos as in-context structured demonstration to help VLA model adapt to OOD tasks.

Problem

• Vision-Language-Action
• out-of-distribution
• generalization
• adaptation
• embodiment transfer

Innovation

• structured demonstration
• in-context learning
• vision-language-action models
• zero-shot adaptation
• embodiment transfer