GRAM: Generalization in Deep RL with a Robust Adaptation Module

2024-12-05

arXiv.org

AI Summary

Deep reinforcement learning faces a critical challenge in real-world deployment: achieving simultaneous in-distribution (ID) adaptation and out-of-distribution (OOD) robustness under unseen dynamics. This work introduces the first unified dynamic generalization framework that jointly models ID adaptability and OOD robustness. Its core innovation is a robust adaptive module capable of detecting and differentially responding to two distinct types of distributional shifts—ID variations and OOD dynamics. Methodologically, we integrate meta-learning with self-supervised dynamics representation learning, yielding a jointly trained architecture comprising an environment encoder, switchable policy heads, and a distribution-shift detection mechanism. Evaluated on multiple quadruped locomotion tasks in simulation, our approach achieves zero-shot cross-dynamics transfer: attaining average success rates of 92% on ID tasks and 78% on OOD tasks—substantially outperforming state-of-the-art baselines.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Intelligent Robots: Learning & Optimization for ROB
• Natural Language Processing: Safety and Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The reliable deployment of deep reinforcement learning in real-world settings requires the ability to generalize across a variety of conditions, including both in-distribution scenarios seen during training as well as novel out-of-distribution scenarios. In this work, we present a framework for dynamics generalization in deep reinforcement learning that unifies these two distinct types of generalization within a single architecture. We introduce a robust adaptation module that provides a mechanism for identifying and reacting to both in-distribution and out-of-distribution environment dynamics, along with a joint training pipeline that combines the goals of in-distribution adaptation and out-of-distribution robustness. Our algorithm GRAM achieves strong generalization performance across in-distribution and out-of-distribution scenarios upon deployment, which we demonstrate on a variety of realistic simulated locomotion tasks with a quadruped robot.

Problem

• Generalizing deep RL across diverse real-world conditions
• Unifying in-distribution and out-of-distribution scenario adaptation
• Enhancing robustness in novel environment dynamics

Innovation

• Unifies in-distribution and out-of-distribution generalization
• Introduces robust adaptation module for dynamics
• Joint training pipeline for adaptation and robustness