On the Saturation Effects of Spectral Algorithms in Large Dimensions

2025-03-01

AI Summary

This paper investigates the saturation phenomenon of spectral algorithms—such as kernel ridge regression and gradient flow—in the high-dimensional regime where $n asymp d^gamma$. Saturation occurs when the true function is overly smooth (i.e., source condition parameter $s$ exceeds a critical threshold), causing the algorithm to fail achieving the information-theoretically optimal rate. We establish, for the first time, that the high-dimensional saturation threshold is $s > au$, contrasting the classical low-dimensional threshold $s > 2 au$. Matching minimax upper and lower bounds are derived. We further reveal a periodic plateau behavior in convergence rates as $gamma$ varies, and identify a fundamental polynomial approximation barrier. Finally, we construct an optimally early-stopped gradient flow estimator that attains information-theoretically optimal generalization up to logarithmic factors.

Technology Category

• Machine Learning: Kernel Methods
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

The saturation effects, which originally refer to the fact that kernel ridge regression (KRR) fails to achieve the information-theoretical lower bound when the regression function is over-smooth, have been observed for almost 20 years and were rigorously proved recently for kernel ridge regression and some other spectral algorithms over a fixed dimensional domain. The main focus of this paper is to explore the saturation effects for a large class of spectral algorithms (including the KRR, gradient descent, etc.) in large dimensional settings where $n asymp d^{gamma}$. More precisely, we first propose an improved minimax lower bound for the kernel regression problem in large dimensional settings and show that the gradient flow with early stopping strategy will result in an estimator achieving this lower bound (up to a logarithmic factor). Similar to the results in KRR, we can further determine the exact convergence rates (both upper and lower bounds) of a large class of (optimal tuned) spectral algorithms with different qualification $ au$'s. In particular, we find that these exact rate curves (varying along $gamma$) exhibit the periodic plateau behavior and the polynomial approximation barrier. Consequently, we can fully depict the saturation effects of the spectral algorithms and reveal a new phenomenon in large dimensional settings (i.e., the saturation effect occurs in large dimensional setting as long as the source condition $s> au$ while it occurs in fixed dimensional setting as long as $s>2 au$).

Problem

• Explores saturation effects in spectral algorithms in high dimensions.
• Proposes improved minimax lower bound for kernel regression in large dimensions.
• Reveals periodic plateau behavior and polynomial approximation barrier in convergence rates.

Innovation

• Improved minimax lower bound for kernel regression
• Gradient flow with early stopping strategy
• Exact convergence rates for spectral algorithms