Dual Alignment Maximin Optimization for Offline Model-based RL

2025-02-02

AI Summary

Offline reinforcement learning (RL) for robot deployment suffers from policy inconsistency and distributional shift due to the sim-to-real gap. To address this, we propose a dual-alignment minimax optimization framework—the first to shift the optimization objective from model fidelity to explicit control of policy divergence. Methodologically, the inner loop enforces policy-trajectory alignment via dual conservative value estimation; the outer loop employs minimax optimization to jointly ensure policy improvement and value estimation consistency while explicitly modeling and correcting offline model bias. Additionally, we embed a self-consistent synthetic data utilization mechanism that balances synthetic-data compatibility with real-environment policy consistency. Evaluated across diverse benchmark tasks, our approach significantly improves deployment stability and generalization, effectively suppressing out-of-distribution (OOD) behaviors and value overestimation.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Online Learning & Bandits

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Offline reinforcement learning agents face significant deployment challenges due to the synthetic-to-real distribution mismatch. While most prior research has focused on improving the fidelity of synthetic sampling and incorporating off-policy mechanisms, the directly integrated paradigm often fails to ensure consistent policy behavior in biased models and underlying environmental dynamics, which inherently arise from discrepancies between behavior and learning policies. In this paper, we first shift the focus from model reliability to policy discrepancies while optimizing for expected returns, and then self-consistently incorporate synthetic data, deriving a novel actor-critic paradigm, Dual Alignment Maximin Optimization (DAMO). It is a unified framework to ensure both model-environment policy consistency and synthetic and offline data compatibility. The inner minimization performs dual conservative value estimation, aligning policies and trajectories to avoid out-of-distribution states and actions, while the outer maximization ensures that policy improvements remain consistent with inner value estimates. Empirical evaluations demonstrate that DAMO effectively ensures model and policy alignments, achieving competitive performance across diverse benchmark tasks.

Problem

• Offline Reinforcement Learning
• Sim-to-Real Transfer
• Policy Consistency

Innovation

• DAMO
• Dual Alignment
• Offline Model-based Reinforcement Learning