R3L: Relative Representations for Reinforcement Learning

2024-04-19

AI Summary

In vision-based reinforcement learning, joint distribution shifts—e.g., between visual appearances (such as seasonal variations) and task objectives (e.g., speed setpoints)—severely impair agent generalization, necessitating frequent retraining. To address this, we propose a relative representation framework that, for the first time, integrates contrastive learning with a learnable mapping network into visual RL. It disentangles encoder embeddings into a universal latent space, enabling independent modeling of visual features and task-specific policies. Our modular agent architecture, coupled with plug-and-play component composition, supports zero-shot adaptation to unseen visual–task pairings. Evaluated in a multi-season, multi-objective autonomous driving simulator, our approach achieves significant gains in zero-shot transfer success rate and reduces retraining frequency by 70%, thereby overcoming the strong environmental specificity inherent in end-to-end paradigms.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Visual Reinforcement Learning is a popular and powerful framework that takes full advantage of the Deep Learning breakthrough. It is known that variations in input domains (e.g., different panorama colors due to seasonal changes) or task domains (e.g., altering the target speed of a car) can disrupt agent performance, necessitating new training for each variation. Recent advancements in the field of representation learning have demonstrated the possibility of combining components from different neural networks to create new models in a zero-shot fashion. In this paper, we build upon relative representations, a framework that maps encoder embeddings to a universal space. We adapt this framework to the Visual Reinforcement Learning setting, allowing to combine agents components to create new agents capable of effectively handling novel visual-task pairs not encountered during training. Our findings highlight the potential for model reuse, significantly reducing the need for retraining and, consequently, the time and computational resources required.

Problem

• Handles novel visual-task pairs
• Reduces need for retraining
• Maps encoder embeddings universally

Innovation

• Relative representations framework
• Zero-shot model combination
• Visual Reinforcement Learning adaptation