Estimating the expected output of wide random MLPs more efficiently than sampling

2026-05-06

AI Summary

This work addresses the challenge of efficiently estimating the expected output of wide random multilayer perceptrons (MLPs) under Gaussian inputs, a task for which conventional sampling-based methods incur prohibitive computational costs—particularly in rare-event probability estimation. To overcome this limitation, the authors propose a sampling-free algorithm that leverages the asymptotic properties of wide networks to analytically propagate activation distributions and compute expected outputs. By integrating cumulant analysis, Hermite polynomial expansions, and Gaussian integration theory, the method achieves accurate estimates without Monte Carlo sampling. Theoretical analysis and empirical experiments demonstrate that the proposed approach significantly reduces FLOPs compared to traditional sampling while attaining comparable mean squared error. Moreover, it outperforms Monte Carlo methods in both rare-event estimation and model training, effectively mitigating catastrophic tail risks.

Abstract

By far the most common way to estimate an expected loss in machine learning is to draw samples, compute the loss on each one, and take the empirical average. However, sampling is not necessarily optimal. Given an MLP at initialization, we show how to estimate its expected output over Gaussian inputs without running samples through the network at all. Instead, we produce approximate representations of the distributions of activations at each layer, leveraging tools such as cumulants and Hermite expansions. We show both theoretically and empirically that for sufficiently wide networks, our estimator achieves a target mean squared error using substantially fewer FLOPs than Monte Carlo sampling. We find moreover that our methods perform particularly well at estimating the probabilities of rare events, and additionally demonstrate how they can be used for model training. Together, these findings suggest a path to producing models with a greatly reduced probability of catastrophic tail risks.

Problem

• expected output estimation
• wide random MLPs
• Gaussian inputs
• rare event probability
• computational efficiency

Innovation

• cumulants
• Hermite expansions
• wide random MLPs
• rare event estimation
• non-sampling expectation