Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design

2026-06-19

AI Summary

This work addresses the lack of non-asymptotic error bounds and slow convergence rates for gradient descent in logistic regression under Gaussian design. Under zero initialization and varying step sizes, the authors establish linear convergence: global linear convergence with small stepsizes and locally faster linear convergence with large stepsizes, ultimately approaching a statistically optimal neighborhood. They provide the first non-asymptotic ℓ₂ error bound of order \( O(\sqrt{\|\theta^*\|_2^5 d / n}) \) and introduce a new estimator achieving the tight high-dimensional rate \( \Theta(\sqrt{\|\theta^*\|_2 d / n}) \). The analysis relies on a covering-and-peeling argument to control gradient bias, refined spectral characterization of the population Hessian, and a newly introduced approximate invertibility condition (AIC). Numerical experiments confirm the sharpness and superiority of the derived error bounds and convergence rates.

Abstract

We consider the parameter estimation problem in logistic regression with Gaussian design: the estimation of a fixed unknown parameter $θ^*\in \mathbb{R}^d$ ($\|θ^*\|_2\ge 1$) from $n$ i.i.d. samples $\{(x_i,y_i)\}_{i=1}^n$, where $x_i\sim N(0,I_d)$ and $y_i|x_i \sim {\rm Bernoulli}(1/(1+\exp(-x_i^\top θ^*)))$. Our main aim is to characterize the finite-sample estimation performance and convergence behavior of gradient descent (GD) on the maximum likelihood objective (i.e., the logistic loss). Under small $O(1)$ stepsize and $0$ initialization, we show that GD linearly converges to a small neighborhood of $θ^*$ achieving an $\ell_2$ error of order $O(\sqrt{\|θ^*\|_2^5d/n})$. This substantially goes beyond existing theoretical results that lack non-asymptotic estimation error rate and exhibit much slower parameter convergence. We also establish a faster local linear convergence to the same statistical error under a large $Θ(\|θ^*\|_2)$ stepsize. The main technical component is to show that the gradient of the logistic loss satisfies a certain approximate invertibility condition (AIC). To that end, we uniformly control the deviation of the gradient from its population counterpart by covering and peeling arguments, and then show that the population GD is a contraction by a delicate analysis based on the eigenvalues of population Hessian matrices. Finally, we build upon the recent work Matsumoto and Mazumdar (2025) and devise a novel efficient estimator that attains a sharper rate in high dimensions. This indicates that the existing non-asymptotic guarantees exhibit sub-optimal dependence on $\|θ^*\|_2$, and that in many regimes $Θ(\sqrt{\|θ^*\|_2d/n})$ is the tight estimation error rate. Numerical examples are provided to corroborate our theoretical results.

Problem

• logistic regression
• gradient descent
• finite-sample analysis
• Gaussian design
• parameter estimation

Innovation

• finite-sample analysis
• gradient descent
• logistic regression
• Gaussian design
• approximate invertibility condition