LoopSR: Looping Sim-and-Real for Lifelong Policy Adaptation of Legged Robots

2024-09-26

arXiv.org

AI Summary

Reinforcement learning policies for legged robots often suffer from performance degradation in real-world deployment due to sim-to-real dynamics mismatch. Method: This paper proposes a simulation-to-reality closed-loop adaptive framework that dynamically reconstructs the simulation environment using real-world trajectory data and enables continual policy optimization. It introduces a novel bidirectional closed-loop mechanism, integrating a Transformer encoder with contrastive learning to model real-world dynamics, while leveraging a VAE for implicit domain representation, domain parameter retrieval, and fusion—enabling lifelong policy adaptation from minimal real-world data. Results: Experiments demonstrate significantly improved data efficiency: the method outperforms strong baselines in both sim-to-sim and sim-to-real transfer tasks, achieving superior generalization and environmental adaptability with only limited real-world trajectories.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement Learning (RL) has shown its remarkable and generalizable capability in legged locomotion through sim-to-real transfer. However, while adaptive methods like domain randomization are expected to make policy more robust to diverse environments, such comprehensiveness potentially detracts from the policy's performance in any specific environment according to the No Free Lunch theorem, leading to a suboptimal solution once deployed in the real world. To address this issue, we propose a lifelong policy adaptation framework named LoopSR, which utilizes a transformer-based encoder to project real-world trajectories into a latent space, and accordingly reconstruct the real-world environments back in simulation for further improvement. Autoencoder architecture and contrastive learning methods are adopted to better extract the characteristics of real-world dynamics. The simulation parameters for continual training are derived by combining predicted parameters from the decoder with retrieved parameters from the simulation trajectory dataset. By leveraging the continual training, LoopSR achieves superior data efficiency compared with strong baselines, with only a limited amount of data to yield eminent performance in both sim-to-sim and sim-to-real experiments.

Problem

• Enhance policy robustness in diverse environments for legged robots
• Address suboptimal real-world deployment of RL policies
• Improve data efficiency in sim-to-real policy adaptation

Innovation

• Transformer-based encoder for real-world trajectory mapping
• Autoencoder and contrastive learning enhance dynamics feature extraction
• Simulated continual training improves data efficiency and performance