PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrations

2025-04-29

AI Summary

Robot policy generalization under few-shot demonstrations remains limited when initial robot and object poses vary; real-robot training is unsafe, and high-fidelity simulation construction is costly. Method: This paper proposes an end-to-end, real-scene-driven sim-to-real reinforcement learning closed-loop framework. It introduces a vision-language model (VLM)-guided projection-relation reward modeling mechanism that enables single-frame image-based object recognition, 6D pose estimation, and 3D model retrieval, automatically generating physically consistent simulation scenes. Policy training follows a two-stage paradigm: VLM-supervised reward pretraining followed by expert demonstration fine-tuning. Contribution/Results: On real robotic arms, the method achieves robust cross-pose deployment with only 3–5 demonstrations. Sim-to-real transfer success rate improves by 42%, and scene construction time decreases by 90%.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Learning from few demonstrations to develop policies robust to variations in robot initial positions and object poses is a problem of significant practical interest in robotics. Compared to imitation learning, which often struggles to generalize from limited samples, reinforcement learning (RL) can autonomously explore to obtain robust behaviors. Training RL agents through direct interaction with the real world is often impractical and unsafe, while building simulation environments requires extensive manual effort, such as designing scenes and crafting task-specific reward functions. To address these challenges, we propose an integrated real-to-sim-to-real pipeline that constructs simulation environments based on expert demonstrations by identifying scene objects from images and retrieving their corresponding 3D models from existing libraries. We introduce a projection-based reward model for RL policy training that is supervised by a vision-language model (VLM) using human-guided object projection relationships as prompts, with the policy further fine-tuned using expert demonstrations. In general, our work focuses on the construction of simulation environments and RL-based policy training, ultimately enabling the deployment of reliable robotic control policies in real-world scenarios.

Problem

• Learning robust policies from few demonstrations
• Building simulation environments without manual effort
• Training RL agents safely and efficiently

Innovation

• Integrated real-to-sim-to-real pipeline for environment construction
• Projection-based reward model supervised by vision-language model
• Policy fine-tuning using expert demonstrations for robustness