Learning Curves and Benign Overfitting of Spectral Algorithms in Large Dimensions

2026-04-25

AI Summary

This work addresses the theoretical gap in characterizing learning behavior and benign overfitting in the under-regularized regime of high-dimensional spectral algorithms. Under a proportional asymptotic setting where sample size and dimensionality grow at the same rate, the authors develop a unified analytical framework for a broad class of kernels satisfying spectral scaling and hypercontractivity conditions, leveraging high-dimensional asymptotics, spectral decomposition of dot-product kernels on the sphere, and source condition modeling. They provide the first complete characterization of the three-phase learning curve—over-regularized, under-regularized, and interpolation regimes—across the entire regularization path, precisely describing the asymptotic excess risk under various source conditions. The analysis reveals that benign overfitting is pervasive in both under-regularized and interpolation regimes, elucidating its underlying mechanism, and extends to kernel ridge regression in high dimensions over ℝᵈ, while also establishing a theoretical link between kernel methods and sequence models.

Technology Category

• Machine Learning: Kernel Methods
• Game Theory and Economic Paradigms: Adversarial Learning
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Existing large-dimensional theory for spectral algorithms resolves either the optimally tuned point or the interpolation limit, but leaves the under-regularized regime unexplored. We study the learning curve and benign overfitting of spectral algorithms in the large-dimensional setting where the sample size and dimension are of comparable order, i.e., $n \asymp d^γ$ for some $γ>0$. We first consider inner-product kernels on the sphere $\mathbb{S}^{d-1}$ and establish a sharp asymptotic characterization of the excess risk across the full regularization path under various source conditions $s \geq 0$, where $s$ measures the relative smoothness of the regression function. Our results reveal that the learning curve is not simply U-shaped but instead consists of three distinct regimes: over-regularized, under-regularized, and interpolation regimes. This characterization allows us to fully capture the benign overfitting phenomenon, demonstrating that benign overfitting arises consistently across both the under-regularized and interpolation regimes whenever $s$ is positive but no larger than a critical threshold. We further show that, in the sufficiently regularized regime, the kernel learning curve is recovered by an associated sequence model. Finally, we extend the learning-curve analysis to large-dimensional KRR for a class of kernels on general domains in $\mathbb{R}^d$ whose low-degree eigenspaces satisfy spectral-scaling and hyper-contractivity conditions.

Problem

• learning curves
• benign overfitting
• spectral algorithms
• large-dimensional setting
• regularization regimes

Innovation

• benign overfitting
• spectral algorithms
• learning curves
• large-dimensional asymptotics
• kernel ridge regression