PRISM: Personalized Robotic Dataset Generation via Image-based Scene and Motion Synthesis

2026-07-06

AI Summary

This work addresses the limited generalization of general-purpose robotic policies in user-specific environments and the high cost and poor scalability of human teleoperation data by introducing PRISM, an end-to-end framework that generates high-fidelity digital twin scenes—semantically and geometrically aligned with rich instance-level diversity—from a single environment image and a natural language instruction, and synthesizes executable task trajectories without human intervention. PRISM is the first approach to enable personalized, scalable robotic training data generation driven jointly by a single image and language instructions, integrating image-guided scene reconstruction, instance-diverse content generation, language-conditioned action synthesis, and trajectory planning. Experiments demonstrate that PRISM significantly outperforms baselines on the LIBERO and LIBERO-Plus benchmarks, achieves 100% success rates across three real-world tasks, and exhibits strong out-of-distribution generalization.

Abstract

Recent advances in large-scale pretrained vision-language-action models have improved robot policy learning, but directly deploying such policies in user-specific environments remains challenging due to limited generalization, which inevitably requires collecting a dataset tailored to the target environment. Teleoperation yields well-aligned data but is costly and difficult to scale, whereas simulation scales easily but struggles to resemble the target environment and generate task-specific trajectories. To meet both simultaneously, we propose PRISM, an end-to-end pipeline that generates personalized robotic datasets from a single image and a natural-language instruction. PRISM constructs digital cousin scenes that are semantically and geometrically aligned with the user environment yet diverse at the instance level, and synthesizes executable demonstrations without human teleoperation. Extensive experiments show that policies trained on PRISM-generated datasets outperform those trained on baseline-generated datasets on LIBERO and LIBERO-Plus, achieve up to 100\% success rate on three real-world manipulation tasks, and maintain stronger performance when evaluated in environments that differ from those seen during training.

Problem

• personalized robotic dataset
• environment generalization
• task-specific trajectories
• data collection
• robot policy learning

Innovation

• personalized dataset generation
• image-based scene synthesis
• motion synthesis
• vision-language-action models
• robot policy learning