Fixing That Free Lunch: When, Where, and Why Synthetic Data Fails in Model-Based Policy Optimization

2025-10-01

AI Summary

This work identifies the core mechanism underlying performance degradation of synthetic data in Model-Based Policy Optimization (MBPO): in high-dimensional continuous control environments such as the DeepMind Control Suite (DMC), mismatched scaling between dynamics and reward models—coupled with suboptimal target state representation—amplifies error propagation and destabilizes model variance during policy optimization. To address this, we propose a scale-adaptive counterfactual action modeling framework with calibrated target state representation, enabling robust synthetic data utilization within MBPO. Empirical evaluation demonstrates that the improved method outperforms Soft Actor-Critic on 5 out of 7 DMC tasks, while retaining high sample efficiency on OpenAI Gym benchmarks. Moreover, it significantly enhances cross-environment generalization and model robustness, validating the efficacy of scale-aware modeling in model-based reinforcement learning.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Adversarial Agents
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Synthetic data is a core component of data-efficient Dyna-style model-based reinforcement learning, yet it can also degrade performance. We study when it helps, where it fails, and why, and we show that addressing the resulting failure modes enables policy improvement that was previously unattainable. We focus on Model-Based Policy Optimization (MBPO), which performs actor and critic updates using synthetic action counterfactuals. Despite reports of strong and generalizable sample-efficiency gains in OpenAI Gym, recent work shows that MBPO often underperforms its model-free counterpart, Soft Actor-Critic (SAC), in the DeepMind Control Suite (DMC). Although both suites involve continuous control with proprioceptive robots, this shift leads to sharp performance losses across seven challenging DMC tasks, with MBPO failing in cases where claims of generalization from Gym would imply success. This reveals how environment-specific assumptions can become implicitly encoded into algorithm design when evaluation is limited. We identify two coupled issues behind these failures: scale mismatches between dynamics and reward models that induce critic underestimation and hinder policy improvement during model-policy coevolution, and a poor choice of target representation that inflates model variance and produces error-prone rollouts. Addressing these failure modes enables policy improvement where none was previously possible, allowing MBPO to outperform SAC in five of seven tasks while preserving the strong performance previously reported in OpenAI Gym. Rather than aiming only for incremental average gains, we hope our findings motivate the community to develop taxonomies that tie MDP task- and environment-level structure to algorithmic failure modes, pursue unified solutions where possible, and clarify how benchmark choices ultimately shape the conditions under which algorithms generalize.

Problem

• Analyzing failure modes of synthetic data in model-based reinforcement learning
• Identifying scale mismatches between dynamics and reward models in MBPO
• Addressing poor target representation that inflates model variance

Innovation

• Identifies scale mismatches between dynamics and reward models
• Addresses poor target representation reducing model variance
• Enables policy improvement in previously failing scenarios