WOMBET: World Model-based Experience Transfer for Robust and Sample-efficient Reinforcement Learning

2026-04-10

AI Summary

This work addresses the poor sample efficiency of reinforcement learning in robotic applications, where data collection is costly and risky. To tackle this challenge, the authors propose the WOMBAT framework, which learns a world model from a source task and generates high-quality offline data via uncertainty-penalized planning. During transfer to a target task, it adaptively fuses offline and online data to ensure stable adaptation. Key innovations include an uncertainty-based trajectory filtering mechanism, a theoretically grounded lower bound on returns, and a finite-sample error decomposition that accounts for both distributional shift and model approximation error. Empirical results demonstrate that WOMBAT significantly outperforms strong baselines on continuous control benchmarks, achieving simultaneous improvements in both sample efficiency and final performance.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Machine Learning: Adversarial Learning & Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Reinforcement learning (RL) in robotics is often limited by the cost and risk of data collection, motivating experience transfer from a source task to a target task. Offline-to-online RL leverages prior data but typically assumes a given fixed dataset and does not address how to generate reliable data for transfer. We propose \textit{World Model-based Experience Transfer} (WOMBET), a framework that jointly generates and utilizes prior data. WOMBET learns a world model in the source task and generates offline data via uncertainty-penalized planning, followed by filtering trajectories with high return and low epistemic uncertainty. It then performs online fine-tuning in the target task using adaptive sampling between offline and online data, enabling a stable transition from prior-driven initialization to task-specific adaptation. We show that the uncertainty-penalized objective provides a lower bound on the true return and derive a finite-sample error decomposition capturing distribution mismatch and approximation error. Empirically, WOMBET improves sample efficiency and final performance over strong baselines on continuous control benchmarks, demonstrating the benefit of jointly optimizing data generation and transfer.

Problem

• experience transfer
• reinforcement learning
• world model
• sample efficiency
• offline-to-online RL

Innovation

• world model
• experience transfer
• uncertainty-penalized planning
• offline-to-online RL
• adaptive sampling