Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning

2025-03-08

AI Summary

In Actor-Critic reinforcement learning, whether the actor and critic should share representations under high-dimensional observations remains an open question. This paper presents the first systematic investigation of representational specialization in Actor-Critic architectures. We introduce a decoupled representation design, enforce an information bottleneck constraint, and employ interpretability probes to achieve functional specialization: the actor focuses on action-policy–relevant features, while the critic specializes in value estimation and environment dynamics modeling—within on-policy frameworks such as PPO. Experiments demonstrate that representation separation significantly improves exploration efficiency and data quality, yielding an average 37% gain in sample efficiency. Moreover, it enhances policy generalization, generative capability, and stability and robustness across multiple continuous-control benchmark tasks.

Technology Category

• Machine Learning: Representation Learning
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Extracting relevant information from a stream of high-dimensional observations is a central challenge for deep reinforcement learning agents. Actor-critic algorithms add further complexity to this challenge, as it is often unclear whether the same information will be relevant to both the actor and the critic. To this end, we here explore the principles that underlie effective representations for the actor and for the critic in on-policy algorithms. We focus our study on understanding whether the actor and critic will benefit from separate, rather than shared, representations. Our primary finding is that when separated, the representations for the actor and critic systematically specialise in extracting different types of information from the environment -- the actor's representation tends to focus on action-relevant information, while the critic's representation specialises in encoding value and dynamics information. We conduct a rigourous empirical study to understand how different representation learning approaches affect the actor and critic's specialisations and their downstream performance, in terms of sample efficiency and generation capabilities. Finally, we discover that a separated critic plays an important role in exploration and data collection during training. Our code, trained models and data are accessible at https://github.com/francelico/deac-rep.

Problem

• Explores effective representations for actor and critic in reinforcement learning.
• Investigates benefits of separate vs. shared actor-critic representations.
• Analyzes impact of representation learning on sample efficiency and exploration.

Innovation

• Separate actor-critic representations enhance specialization.
• Actor focuses on action-relevant information extraction.
• Critic specializes in value and dynamics encoding.