Robust Adversarial Policy Optimization Under Dynamics Uncertainty

2026-04-13

AI Summary

This work addresses the significant performance degradation of reinforcement learning policies when training and testing environment dynamics differ, a setting where existing methods struggle to balance robustness and performance. The authors propose a dual framework that operates at two levels: at the trajectory level, an adversarial network approximates a temperature parameter to enable efficient worst-case experience replay; at the model level, Boltzmann reweighting prioritizes dynamics models that are more challenging for the current policy. This approach is the first to directly leverage duality to characterize the robustness–performance trade-off, combining trajectory-level adversarial control with model-level policy-sensitive sampling to effectively avoid over-conservatism and instability. Experiments demonstrate that the method significantly outperforms existing robust RL baselines while preserving dual tractability, exhibiting superior generalization and stability under out-of-distribution dynamics.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Reinforcement learning (RL) policies often fail under dynamics that differ from training, a gap not fully addressed by domain randomization or existing adversarial RL methods. Distributionally robust RL provides a formal remedy but still relies on surrogate adversaries to approximate intractable primal problems, leaving blind spots that potentially cause instability and over-conservatism. We propose a dual formulation that directly exposes the robustness-performance trade-off. At the trajectory level, a temperature parameter from the dual problem is approximated with an adversarial network, yielding efficient and stable worst-case rollouts within a divergence bound. At the model level, we employ Boltzmann reweighting over dynamics ensembles, focusing on more adverse environments to the current policy rather than uniform sampling. The two components act independently and complement each other: trajectory-level steering ensures robust rollouts, while model-level sampling provides policy-sensitive coverage of adverse dynamics. The resulting framework, robust adversarial policy optimization (RAPO) outperforms robust RL baselines, improving resilience to uncertainty and generalization to out-of-distribution dynamics while maintaining dual tractability.

Problem

• dynamics uncertainty
• robust reinforcement learning
• adversarial policy
• distributional robustness
• out-of-distribution generalization

Innovation

• distributionally robust reinforcement learning
• dual formulation
• adversarial policy optimization
• Boltzmann reweighting
• dynamics uncertainty