Harmful Overfitting in Sobolev Spaces

2026-01-31

AI Summary

This work investigates the generalization behavior of norm-minimizing interpolants in Sobolev spaces under noisy data, revealing a persistent and non-vanishing generalization error—referred to as benign overfitting—even in the large-sample regime. By introducing geometric arguments combined with Sobolev inequalities, the analysis is extended for the first time from Hilbert spaces (corresponding to \( p = 2 \)) to general Sobolev spaces with arbitrary \( p \in [1, \infty) \). The study identifies harmful neighborhoods near training points where interpolation amplifies noise. Under assumptions on label noise and data distribution regularity, it is shown that the generalization error of smoothness-preferring interpolants is, with high probability, bounded below by a positive constant. This underscores the critical role of function space selection in determining generalization performance.

Technology Category

• Machine Learning: Learning with Manifolds
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Motivated by recent work on benign overfitting in overparameterized machine learning, we study the generalization behavior of functions in Sobolev spaces $W^{k, p}(\mathbb{R}^d)$ that perfectly fit a noisy training data set. Under assumptions of label noise and sufficient regularity in the data distribution, we show that approximately norm-minimizing interpolators, which are canonical solutions selected by smoothness bias, exhibit harmful overfitting: even as the training sample size $n \to \infty$, the generalization error remains bounded below by a positive constant with high probability. Our results hold for arbitrary values of $p \in [1, \infty)$, in contrast to prior results studying the Hilbert space case ($p = 2$) using kernel methods. Our proof uses a geometric argument which identifies harmful neighborhoods of the training data using Sobolev inequalities.

Problem

• harmful overfitting
• Sobolev spaces
• generalization error
• interpolation
• label noise