Scaling Effects and Uncertainty Quantification in Neural Actor Critic Algorithms

2026-01-25

AI Summary

This work addresses the lack of systematic analysis regarding the convergence behavior and output uncertainty of neural Actor-Critic algorithms across varying network widths. Focusing on shallow neural network implementations, the study introduces a tunable inverse power-law width scaling parameter ranging between 1/2 and 1. By integrating asymptotic expansions, stochastic differential equations, and statistical estimation theory, the authors characterize the asymptotic statistical properties of the algorithm’s output and quantify its approximation error. Theoretical analysis demonstrates that the output variance decays at a rate proportional to the network width raised to the power of (1/2 − scaling exponent), revealing a direct influence of this exponent on statistical robustness. Building on these insights, a theory-driven criterion for hyperparameter selection is proposed, and numerical experiments confirm that setting the scaling parameter close to 1 significantly enhances both convergence speed and stability.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Online Learning & Bandits
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Economics, Online Markets and Human Computation: Social networks and social learning
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Social Networks and Social Media: Computational social science

Abstract

We investigate the neural Actor Critic algorithm using shallow neural networks for both the Actor and Critic models. The focus of this work is twofold: first, to compare the convergence properties of the network outputs under various scaling schemes as the network width and the number of training steps tend to infinity; and second, to provide precise control of the approximation error associated with each scaling regime. Previous work has shown convergence to ordinary differential equations with random initial conditions under inverse square root scaling in the network width. In this work, we shift the focus from convergence speed alone to a more comprehensive statistical characterization of the algorithm's output, with the goal of quantifying uncertainty in neural Actor Critic methods. Specifically, we study a general inverse polynomial scaling in the network width, with an exponent treated as a tunable hyperparameter taking values strictly between one half and one. We derive an asymptotic expansion of the network outputs, interpreted as statistical estimators, in order to clarify their structure. To leading order, we show that the variance decays as a power of the network width, with an exponent equal to one half minus the scaling parameter, implying improved statistical robustness as the scaling parameter approaches one. Numerical experiments support this behavior and further suggest faster convergence for this choice of scaling. Finally, our analysis yields concrete guidelines for selecting algorithmic hyperparameters, including learning rates and exploration rates, as functions of the network width and the scaling parameter, ensuring provably favorable statistical behavior.

Problem

• Uncertainty Quantification
• Scaling Effects
• Neural Actor Critic
• Statistical Robustness
• Asymptotic Analysis

Innovation

• neural Actor-Critic
• scaling laws
• uncertainty quantification
• asymptotic expansion
• statistical robustness