Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning

2025-02-04

AI Summary

To address low sample efficiency and high zero-shot failure rates in sim-to-real policy transfer for force-sensitive dexterous manipulation, this paper proposes the Simulation-Guided Fine-Tuning (SGFT) framework. Its core innovation is the first explicit use of a simulation-trained value function as a structural prior to guide real-robot exploration and policy update. SGFT integrates reinforcement learning fine-tuning, cross-domain value function transfer, and policy-guided exploration. Evaluated on PyBullet/MuJoCo simulators and Shadow Hand/DexArm real-world platforms, SGFT achieves successful adaptation across five high-precision dexterous manipulation tasks—even when zero-shot transfer completely fails. It reduces real-world interaction samples by up to 90% over baselines and, for the first time, converges on previously intractable force-controlled tasks. Theoretical analysis further establishes its robustness under large dynamics discrepancies between simulation and reality.

Technology Category

• Intelligent Robots: Manipulation
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Robot learning requires a considerable amount of high-quality data to realize the promise of generalization. However, large data sets are costly to collect in the real world. Physics simulators can cheaply generate vast data sets with broad coverage over states, actions, and environments. However, physics engines are fundamentally misspecified approximations to reality. This makes direct zero-shot transfer from simulation to reality challenging, especially in tasks where precise and force-sensitive manipulation is necessary. Thus, fine-tuning these policies with small real-world data sets is an appealing pathway for scaling robot learning. However, current reinforcement learning fine-tuning frameworks leverage general, unstructured exploration strategies which are too inefficient to make real-world adaptation practical. This paper introduces the Simulation-Guided Fine-tuning (SGFT) framework, which demonstrates how to extract structural priors from physics simulators to substantially accelerate real-world adaptation. Specifically, our approach uses a value function learned in simulation to guide real-world exploration. We demonstrate this approach across five real-world dexterous manipulation tasks where zero-shot sim-to-real transfer fails. We further demonstrate our framework substantially outperforms baseline fine-tuning methods, requiring up to an order of magnitude fewer real-world samples and succeeding at difficult tasks where prior approaches fail entirely. Last but not least, we provide theoretical justification for this new paradigm which underpins how SGFT can rapidly learn high-performance policies in the face of large sim-to-real dynamics gaps. Project webpage: https://weirdlabuw.github.io/sgft/{weirdlabuw.github.io/sgft}

Problem

• Overcome sim-to-real transfer challenges
• Accelerate real-world policy adaptation
• Reduce real-world data requirements

Innovation

• Simulation-Guided Fine-tuning framework
• Value function from simulation
• Accelerates real-world adaptation