Linear Mixture Distributionally Robust Markov Decision Processes

2025-05-23

AI Summary

Real-world decision-making often suffers from *off-dynamics*—a performance collapse of policies trained in a source domain when deployed in a target domain with differing state transitions. Existing distributionally robust MDPs (DRMDPs) rely on prior-driven uncertainty sets, which are coarse-grained and constrained by restrictive *(s,a)*- or *d*-rectangularity assumptions. To address this, we propose a novel *linear mixture DRMDP* framework that shifts uncertainty modeling from the transition kernel space to the mixing weight parameter space. This is the first work to embed distributional robustness directly into linear mixture dynamics priors, thereby breaking rectangularity constraints and enabling tighter, physically interpretable characterizations of dynamic shifts. Leveraging *f*-divergence, we construct a general uncertainty set, derive analytical robust value functions, design a meta-level robust policy optimization algorithm, and establish optimal sample complexity bounds under total variation, KL, and χ² divergences. Our analysis establishes statistical learnability, offering a new paradigm for distributionally robust reinforcement learning.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Reasoning under Uncertainty: Stochastic Optimization
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Many real-world decision-making problems face the off-dynamics challenge: the agent learns a policy in a source domain and deploys it in a target domain with different state transitions. The distributionally robust Markov decision process (DRMDP) addresses this challenge by finding a robust policy that performs well under the worst-case environment within a pre-specified uncertainty set of transition dynamics. Its effectiveness heavily hinges on the proper design of these uncertainty sets, based on prior knowledge of the dynamics. In this work, we propose a novel linear mixture DRMDP framework, where the nominal dynamics is assumed to be a linear mixture model. In contrast with existing uncertainty sets directly defined as a ball centered around the nominal kernel, linear mixture DRMDPs define the uncertainty sets based on a ball around the mixture weighting parameter. We show that this new framework provides a more refined representation of uncertainties compared to conventional models based on $(s,a)$-rectangularity and $d$-rectangularity, when prior knowledge about the mixture model is present. We propose a meta algorithm for robust policy learning in linear mixture DRMDPs with general $f$-divergence defined uncertainty sets, and analyze its sample complexities under three divergence metrics instantiations: total variation, Kullback-Leibler, and $chi^2$ divergences. These results establish the statistical learnability of linear mixture DRMDPs, laying the theoretical foundation for future research on this new setting.

Problem

• Addresses off-dynamics challenge in decision-making across domains
• Proposes linear mixture DRMDP for refined uncertainty representation
• Analyzes sample complexities under various divergence metrics

Innovation

• Linear mixture model for nominal dynamics
• Uncertainty sets based on mixture parameters
• Meta algorithm for robust policy learning