Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation

2026-05-04

AI Summary

Simulated videos often suffer from large domain gaps and insufficient environmental diversity, hindering the effective training of vision-language-action (VLA) models with strong real-world generalization. To address this, this work proposes an efficient video augmentation framework that first extracts semantic segmentation masks and captions from simulated videos as structured conditions, then rewrites captions to enrich environmental descriptions. A conditional video diffusion model subsequently generates photorealistic videos while preserving task semantics and action trajectories. The approach further incorporates a diffusion feature reuse mechanism to accelerate generation and employs a coreset sampling strategy to select a non-redundant subset for augmentation under limited computational budgets. Evaluated on Robotwin 2.0, LIBERO, LIBERO-Plus, and real robotic platforms, the method consistently yields significant performance gains—e.g., +8% for RDT-1B on Robotwin 2.0 and +5.1% for π₀ on LIBERO-Plus.

Abstract

Vision-language-action (VLA) models typically rely on large-scale real-world videos, whereas simulated data, despite being inexpensive and highly parallelizable to collect, often suffers from a substantial visual domain gap and limited environmental diversity, resulting in weak real-world generalization. We present an efficient video augmentation framework that converts simulated VLA videos into realistic training videos while preserving task semantics and action trajectories. Our pipeline extracts structured conditions from simulation via video semantic segmentation and video captioning, rewrites captions to diversify environments, and uses a conditional video transfer model to synthesize realistic videos. To make augmentation practical at scale, we introduce a diffusion feature-reuse mechanism that reuses video tokens across adjacent timesteps to accelerate generation, and a coreset sampling strategy that identifies a compact, non-redundant subset for augmentation under limited computation. Extensive experiments on Robotwin 2.0, LIBERO, LIBERO-Plus, and a real robotic platform demonstrate consistent improvements. For example, our method improves RDT-1B by 8% on Robotwin 2.0, and boosts $π_0$ by 5.1% on the more challenging LIBERO-Plus benchmark. Code is available at: https://github.com/nanfangxiansheng/Seeing-Realism-from-Simulation.

Problem

• vision-language-action
• simulation-to-reality gap
• video augmentation
• domain adaptation
• real-world generalization

Innovation

• video transfer
• simulation-to-reality
• diffusion feature reuse
• coreset sampling
• vision-language-action