Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access

2025-09-30

AI Summary

To address high value estimation bias and low policy learning efficiency in partially observable reinforcement learning, this paper proposes a novel asymmetric actor-critic framework: during training, the critic is enhanced with limited privileged signals—requiring no full state access—to improve policy gradient estimation quality. We theoretically prove that the resulting policy gradient remains unbiased. Furthermore, we introduce a kernel-based, return-prediction-error-driven metric to quantify the effectiveness of privileged signals and integrate it with function approximation for robust value estimation. Experiments on navigation benchmarks and synthetic partially observable environments demonstrate significant improvements: +32% average sample efficiency and 41% reduction in value function mean squared error (MSE). This work provides the first empirical validation that *incomplete* privileged information yields substantial gains in asymmetric RL.

Technology Category

• Game Theory and Economic Paradigms: Adversarial Learning
• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits

Application Category

• Security and Privacy: Privacy-enhancing technologies
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Reinforcement learning in partially observable environments requires agents to act under uncertainty from noisy, incomplete observations. Asymmetric actor-critic methods leverage privileged information during training to improve learning under these conditions. However, existing approaches typically assume full-state access during training. In this work, we challenge this assumption by proposing a novel actor-critic framework, called informed asymmetric actor-critic, that enables conditioning the critic on arbitrary privileged signals without requiring access to the full state. We show that policy gradients remain unbiased under this formulation, extending the theoretical foundation of asymmetric methods to the more general case of privileged partial information. To quantify the impact of such signals, we propose informativeness measures based on kernel methods and return prediction error, providing practical tools for evaluating training-time signals. We validate our approach empirically on benchmark navigation tasks and synthetic partially observable environments, showing that our informed asymmetric method improves learning efficiency and value estimation when informative privileged inputs are available. Our findings challenge the necessity of full-state access and open new directions for designing asymmetric reinforcement learning methods that are both practical and theoretically sound.

Problem

• Extends asymmetric actor-critic methods beyond full-state access assumptions
• Enables critic conditioning on arbitrary privileged signals during training
• Quantifies signal impact through informativeness measures and empirical validation

Innovation

• Critic conditions on arbitrary privileged signals
• Policy gradients remain unbiased with partial information
• Informativeness measures evaluate training-time signals