A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models

2025-02-18

AI Summary

Deep reinforcement learning (DRL) policies trained in simulation often suffer from performance degradation and safety risks when deployed in the real world—commonly termed the sim-to-real gap. Method: This paper proposes the first unified taxonomy for sim-to-real transfer, grounded in the four core components of Markov Decision Processes (MDPs): states, actions, transitions, and rewards. It systematically organizes existing techniques—including domain adaptation, representation alignment, simulation modeling, and foundation-model–guided policy transfer (e.g., LLMs and multimodal models)—along this MDP-centric axis. Contribution/Results: We introduce the first openly maintained knowledge graph and reproducible benchmark platform for sim-to-real research, featuring over 100 algorithms and standardized evaluation protocols. Our analysis identifies six fundamental open challenges and uncovers novel pathways for leveraging foundation models to bridge the sim-to-real gap, thereby providing both theoretical foundations and practical paradigms for robust cross-domain policy transfer.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Deep Reinforcement Learning (RL) has been explored and verified to be effective in solving decision-making tasks in various domains, such as robotics, transportation, recommender systems, etc. It learns from the interaction with environments and updates the policy using the collected experience. However, due to the limited real-world data and unbearable consequences of taking detrimental actions, the learning of RL policy is mainly restricted within the simulators. This practice guarantees safety in learning but introduces an inevitable sim-to-real gap in terms of deployment, thus causing degraded performance and risks in execution. There are attempts to solve the sim-to-real problems from different domains with various techniques, especially in the era with emerging techniques such as large foundations or language models that have cast light on the sim-to-real. This survey paper, to the best of our knowledge, is the first taxonomy that formally frames the sim-to-real techniques from key elements of the Markov Decision Process (State, Action, Transition, and Reward). Based on the framework, we cover comprehensive literature from the classic to the most advanced methods including the sim-to-real techniques empowered by foundation models, and we also discuss the specialties that are worth attention in different domains of sim-to-real problems. Then we summarize the formal evaluation process of sim-to-real performance with accessible code or benchmarks. The challenges and opportunities are also presented to encourage future exploration of this direction. We are actively maintaining a to include the most up-to-date sim-to-real research outcomes to help the researchers in their work.

Problem

• Sim-to-real gap in RL deployment
• Enhancing RL with foundation models
• Formal evaluation of sim-to-real performance

Innovation

• Sim-to-Real gap reduction
• Foundation models integration
• Markov Decision Process framework