Hyperspherical Normalization for Scalable Deep Reinforcement Learning

2025-02-21

AI Summary

In deep reinforcement learning (DRL), scaling model and computational capacity often exacerbates training instability and overfitting due to non-stationary data distributions. To address this, we propose SimbaV2—a novel architecture that introduces hyperspherical normalization to DRL for the first time, jointly integrating distributional value estimation with reward-adaptive scaling to co-constrain weight and feature norms, thereby significantly improving gradient stability. Our key contributions are: (1) a norm-constrained optimization framework driven by hyperspherical normalization, ensuring bounded parameter and activation magnitudes; and (2) a distributed value estimation method with dynamic reward scaling, effectively mitigating scalability bottlenecks under non-stationary data regimes. Evaluated across four domains and 57 continuous control benchmarks, SimbaV2 consistently surpasses state-of-the-art methods, delivering substantial gains in both training stability and generalization performance for large-scale DRL models.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Scaling up the model size and computation has brought consistent performance improvements in supervised learning. However, this lesson often fails to apply to reinforcement learning (RL) because training the model on non-stationary data easily leads to overfitting and unstable optimization. In response, we introduce SimbaV2, a novel RL architecture designed to stabilize optimization by (i) constraining the growth of weight and feature norm by hyperspherical normalization; and (ii) using a distributional value estimation with reward scaling to maintain stable gradients under varying reward magnitudes. Using the soft actor-critic as a base algorithm, SimbaV2 scales up effectively with larger models and greater compute, achieving state-of-the-art performance on 57 continuous control tasks across 4 domains. The code is available at https://dojeon-ai.github.io/SimbaV2.

Problem

• Stabilize optimization in reinforcement learning
• Prevent overfitting in large RL models
• Maintain stable gradients with varying rewards

Innovation

• Hyperspherical normalization constrains weight growth
• Distributional value estimation stabilizes gradients
• Scalable reinforcement learning with larger models