Generalisation and benign over-fitting for linear regression onto random functional covariates

2025-08-19

AI Summary

This paper investigates the predictive performance of ridge and ordinary least squares regression when covariates are generated by evaluating $p$ mean-square continuous functions—defined on a latent metric space—at random locations, plus additive noise. Under this setting, covariate vectors are exchangeable but not i.i.d., better reflecting real-world functional data structures. Methodologically, the authors depart from classical i.i.d. assumptions and develop a novel generalization error analysis framework tailored to random functional covariates. Leveraging fourth-moment conditions, inter-dimensional independence, and the Barzilai–Borwein (Barzilai–Shamir) step-size theory, they derive probabilistic bounds and convergence rates for the excess prediction risk. A key contribution is the first identification of covariate noise as a critical modulator of benign overfitting. For regimes where $p$ grows rapidly with sample size $n$, they obtain explicit convergence rates and quantify how model complexity, noise magnitude, and sample size jointly govern generalization behavior.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study theoretical predictive performance of ridge and ridge-less least-squares regression when covariate vectors arise from evaluating $p$ random, means-square continuous functions over a latent metric space at $n$ random and unobserved locations, subject to additive noise. This leads us away from the standard assumption of i.i.d. data to a setting in which the $n$ covariate vectors are exchangeable but not independent in general. Under an assumption of independence across dimensions, $4$-th order moment, and other regularity conditions, we obtain probabilistic bounds on a notion of predictive excess risk adapted to our random functional covariate setting, making use of recent results of Barzilai and Shamir. We derive convergence rates in regimes where $p$ grows suitably fast relative to $n$, illustrating interplay between ingredients of the model in determining convergence behaviour and the role of additive covariate noise in benign-overfitting.

Problem

• Studying predictive performance of ridge regression on functional covariates
• Analyzing generalization under exchangeable but non-i.i.d. covariate vectors
• Investigating benign overfitting with additive noise in high dimensions

Innovation

• Ridge regression on functional covariates
• Exchangeable non-i.i.d. data analysis
• Probabilistic bounds for predictive risk