Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning

2025-05-19

AI Summary

Offline model-based reinforcement learning faces two key challenges: (1) misalignment between world model training objectives and policy optimization goals, and (2) insufficient robustness to environmental perturbations—such as adversarial noise—during deployment. To address these, we propose a policy-driven dynamic world model adaptation framework that unifies world model learning and policy optimization under a robustness-oriented minimax objective—the first such formulation. We formulate the learning dynamics as a Stackelberg game and provide theoretical convergence guarantees. Our method integrates adversarial robust training with implicit world model modeling. Empirically, it achieves state-of-the-art performance on 12 noisy D4RL MuJoCo tasks and 3 stochastic tokamak control tasks, demonstrating significantly improved policy stability and generalization under perturbations.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Offline reinforcement learning (RL) offers a powerful paradigm for data-driven control. Compared to model-free approaches, offline model-based RL (MBRL) explicitly learns a world model from a static dataset and uses it as a surrogate simulator, improving data efficiency and enabling potential generalization beyond the dataset support. However, most existing offline MBRL methods follow a two-stage training procedure: first learning a world model by maximizing the likelihood of the observed transitions, then optimizing a policy to maximize its expected return under the learned model. This objective mismatch results in a world model that is not necessarily optimized for effective policy learning. Moreover, we observe that policies learned via offline MBRL often lack robustness during deployment, and small adversarial noise in the environment can lead to significant performance degradation. To address these, we propose a framework that dynamically adapts the world model alongside the policy under a unified learning objective aimed at improving robustness. At the core of our method is a maximin optimization problem, which we solve by innovatively utilizing Stackelberg learning dynamics. We provide theoretical analysis to support our design and introduce computationally efficient implementations. We benchmark our algorithm on twelve noisy D4RL MuJoCo tasks and three stochastic Tokamak Control tasks, demonstrating its state-of-the-art performance.

Problem

• Addresses objective mismatch in offline MBRL world model training
• Improves robustness of policies against adversarial noise
• Unifies world model and policy learning for better performance

Innovation

• Dynamic world model adaptation with policy learning
• Maximin optimization using Stackelberg learning dynamics
• Unified objective for robustness and performance