DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptation

2026-08-29

AI Summary

DREAM框架通过重建工作空间、自动翻译指令并生成机器人轨迹来为预训练的VLA模型提供微调数据，解决了在新环境中收集标注数据成本高的问题。

Abstract

Vision-language-action (VLA) models have made strong progress in language-conditioned robot manipulation, but improving their performance in a new workspace still often requires action-labeled data from that environment. Collecting such data by human teleoperation is costly, especially when each workspace, object arrangement, or task may require new demonstrations. We present DREAM, a framework that generates fine-tuning data for a pretrained VLA from a captured workspace and a language instruction, without requiring a task-specific human demonstration. DREAM reconstructs the workspace, automatically translates the instruction into symbolic task goals and success criteria using a large language model, and uses task-and-motion planning to generate feasible robot trajectories. The planned trajectories are augmented across randomized object configurations, verified by the generated success criteria, and rendered into image-action examples for VLA fine-tuning. Through real-robot experiments on language-conditioned manipulation tasks, we study whether DREAM can serve as a scalable data-collection system for the deployment workspace by examining whether fine-tuning on its automatically generated data improves success over direct deployment and how its data-collection cost compares with human teleoperation when adapting a VLA to a new workspace.

Problem

• Vision-language-action models
• language-conditioned robot manipulation
• action-labeled data
• human teleoperation
• workspace adaptation

Innovation

• Real-to-Sim
• Scalable Policy Adaptation
• Vision-Language-Action Models
• Automatic Demonstration Generation
• Task-and-Motion Planning