Optimal Implicit Bias in Linear Regression

2025-06-20

AI Summary

This work investigates the impact of implicit bias induced by optimization algorithms on generalization in overparameterized linear regression, addressing the fundamental question: *Which implicit bias yields optimal generalization?* Methodologically, we integrate asymptotic analysis under non-isotropic Gaussian covariates, minimization of convex potential functions, spectral characterization of the covariance matrix, and log-concavity analysis of Gaussian-prior convolutions. Our main contribution is the first tight lower bound on the generalization error for convex-implicit-bias interpolators, along with a precise characterization of the necessary and sufficient conditions for achievability. The bound explicitly depends on the overparameterization ratio, noise variance, eigenvalue spectrum of the feature covariance, and the prior distribution of the true parameter. Moreover, under mild regularity conditions, we construct an explicit convex implicit bias that achieves this bound—thereby revealing an intrinsic connection between optimal generalization, data geometry, and prior knowledge.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Most modern learning problems are over-parameterized, where the number of learnable parameters is much greater than the number of training data points. In this over-parameterized regime, the training loss typically has infinitely many global optima that completely interpolate the data with varying generalization performance. The particular global optimum we converge to depends on the implicit bias of the optimization algorithm. The question we address in this paper is, ``What is the implicit bias that leads to the best generalization performance?". To find the optimal implicit bias, we provide a precise asymptotic analysis of the generalization performance of interpolators obtained from the minimization of convex functions/potentials for over-parameterized linear regression with non-isotropic Gaussian data. In particular, we obtain a tight lower bound on the best generalization error possible among this class of interpolators in terms of the over-parameterization ratio, the variance of the noise in the labels, the eigenspectrum of the data covariance, and the underlying distribution of the parameter to be estimated. Finally, we find the optimal convex implicit bias that achieves this lower bound under certain sufficient conditions involving the log-concavity of the distribution of a Gaussian convolved with the prior of the true underlying parameter.

Problem

• Identify optimal implicit bias for best generalization in over-parameterized linear regression
• Analyze generalization performance of interpolators in non-isotropic Gaussian data settings
• Determine sufficient conditions for achieving tight lower bound on generalization error

Innovation

• Analyzes generalization in over-parameterized linear regression
• Derives tight lower bound for generalization error
• Identifies optimal convex implicit bias conditions