Stable Deep Reinforcement Learning via Isotropic Gaussian Representations

2026-02-22

AI Summary

In non-stationary environments, deep reinforcement learning often suffers from representation degradation and training instability due to time-varying targets and shifting data distributions. This work proposes a lightweight and efficient sketch-based isotropic Gaussian regularization method that encourages the agent to learn isotropic Gaussian representations. Theoretical analysis demonstrates that such representations offer several advantages, including stable tracking of dynamic targets, maximal entropy, and balanced utilization of embedding dimensions. By integrating a linear readout structure within a deep reinforcement learning framework, the proposed approach significantly enhances performance across a variety of tasks while effectively mitigating representation collapse, neuron dormancy, and training instability.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Deep reinforcement learning systems often suffer from unstable training dynamics due to non-stationarity, where learning objectives and data distributions evolve over time. We show that under non-stationary targets, isotropic Gaussian embeddings are provably advantageous. In particular, they induce stable tracking of time-varying targets for linear readouts, achieve maximal entropy under a fixed variance budget, and encourage a balanced use of all representational dimensions--all of which enable agents to be more adaptive and stable. Building on this insight, we propose the use of Sketched Isotropic Gaussian Regularization for shaping representations toward an isotropic Gaussian distribution during training. We demonstrate empirically, over a variety of domains, that this simple and computationally inexpensive method improves performance under non-stationarity while reducing representation collapse, neuron dormancy, and training instability.

Problem

• non-stationarity
• training instability
• representation collapse
• deep reinforcement learning

Innovation

• Isotropic Gaussian Representations
• Non-stationarity
• Representation Regularization
• Deep Reinforcement Learning
• Stable Training