ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertainty

2025-10-13

AI Summary

To address the excessive conservatism and high computational cost of robust reinforcement learning (RRL) policies under environmental dynamics with epistemic uncertainty, this paper proposes an adaptive-rank bilevel optimization framework. Methodologically, it replaces conventional nested min-max optimization with a Wasserstein ambiguity set to model dynamic uncertainty, and jointly optimizes policy parameters and rank dimension under a fixed-rank constraint—leveraging low-rank manifold projection and a learnable policy rank. Guided by theory, the framework aligns policy complexity with the task’s intrinsic dimensionality, enabling non-overparameterized robust learning. Evaluated on the MuJoCo benchmark, the method significantly outperforms baselines including SAC and RNAC, converging to the task’s intrinsic rank while achieving superior performance and improved computational efficiency.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Robust reinforcement learning (Robust RL) seeks to handle epistemic uncertainty in environment dynamics, but existing approaches often rely on nested min--max optimization, which is computationally expensive and yields overly conservative policies. We propose extbf{Adaptive Rank Representation (AdaRL)}, a bi-level optimization framework that improves robustness by aligning policy complexity with the intrinsic dimension of the task. At the lower level, AdaRL performs policy optimization under fixed-rank constraints with dynamics sampled from a Wasserstein ball around a centroid model. At the upper level, it adaptively adjusts the rank to balance the bias--variance trade-off, projecting policy parameters onto a low-rank manifold. This design avoids solving adversarial worst-case dynamics while ensuring robustness without over-parameterization. Empirical results on MuJoCo continuous control benchmarks demonstrate that AdaRL not only consistently outperforms fixed-rank baselines (e.g., SAC) and state-of-the-art robust RL methods (e.g., RNAC, Parseval), but also converges toward the intrinsic rank of the underlying tasks. These results highlight that adaptive low-rank policy representations provide an efficient and principled alternative for robust RL under model uncertainty.

Problem

• Robust RL handles epistemic uncertainty in environment dynamics
• Existing methods use costly min-max optimization causing conservative policies
• AdaRL adapts policy complexity to task's intrinsic dimension for robustness

Innovation

• Adaptive Rank Representation balances bias-variance trade-off
• Bi-level optimization aligns policy complexity with task dimension
• Low-rank manifold projection avoids adversarial worst-case dynamics