Scalable Statistical Inference in Stochastic Gradient Descent

2026-08-31

AI Summary

本文针对高维条件下传统方法无法有效估计SGD置信区间的问题，提出了一种基于等批量均值法和t-分布的统计框架，结合Quasi-Monte Carlo方法和Lugsail方差估计器，实现了高效且可扩展的统计推断。

Abstract

Constructing confidence regions for stochastic gradient descent (SGD) ideally requires estimating the asymptotic covariance matrix, a severe computational bottleneck in high dimensions. Traditional cancellation-based batch means methods bypass this estimation but require inverting a sample batch covariance matrix. This introduces strict mathematical degeneracy when the parameter dimension exceeds the number of batches. To address this problem, we utilize equal batch size batch means method and propose a simultaneous, marginal-friendly framework. The proposed marginal statistics has a asymptotic Student's $t$-distribution, and eliminates the matrix inversion step, entirely circumventing high-dimensional degeneracy. To achieve valid simultaneous coverage, we present an algorithm utilizing wild bootstrap samples drawn from a statistic as a function of only the diagonals of the variance-covariance estimator, and to further incorporate the contribution of cross-dependencies, we introduce an efficient Quasi-Monte Carlo procedure utilizing a $t$-copula approximation. Additionally, we integrate a Lugsail variance estimator to aggressively correct finite-sample bias and under-coverage. The proposed methodology delivers interpretable, simultaneous hyper-rectangular confidence regions that are statistically robust, memory-efficient, and strictly scalable for high-dimensional inference. The theoretical results are supported by extensive numerical simulation analysis through various aspects of dimension, number of batches and error structure.

Problem

• Stochastic Gradient Descent
• Confidence Regions
• Asymptotic Covariance Matrix
• High-dimensional Inference
• Batch Means

Innovation

• marginal-friendly framework
• wild bootstrap samples
• Quasi-Monte Carlo procedure
• t-copula approximation
• Lugsail variance estimator