Overfitting Regimes of Nadaraya-Watson Interpolators

2025-02-11

AI Summary

This work investigates the generalization behavior of the Nadaraya–Watson (NW) estimator under noisy data, revealing a bandwidth-dependent non-monotonic overfitting spectrum: catastrophic → benign → mild. Methodologically, we employ nonparametric kernel regression analysis, asymptotic statistical derivations, and bandwidth sensitivity modeling to rigorously characterize the phase transitions governing overfitting regimes. We establish, for the first time, precise theoretical boundary conditions distinguishing these three overfitting phases. Comprehensive numerical experiments fully reproduce and validate all theoretical predictions. Our results challenge the conventional wisdom that overfitting inevitably harms generalization, formally establishing the “benign overfitting” paradigm within interpolation learning. By providing an analytically tractable benchmark, this work offers novel insights into the generalization mechanisms of modern overparameterized models.

Technology Category

• Machine Learning: Kernel Methods
• Game Theory and Economic Paradigms: Adversarial Learning
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In recent years, there has been much interest in understanding the generalization behavior of interpolating predictors, which overfit on noisy training data. Whereas standard analyses are concerned with whether a method is consistent or not, recent observations have shown that even inconsistent predictors can generalize well. In this work, we revisit the classic interpolating Nadaraya-Watson (NW) estimator (also known as Shepard's method), and study its generalization capabilities through this modern viewpoint. In particular, by varying a single bandwidth-like hyperparameter, we prove the existence of multiple overfitting behaviors, ranging non-monotonically from catastrophic, through benign, to tempered. Our results highlight how even classical interpolating methods can exhibit intricate generalization behaviors. Numerical experiments complement our theory, demonstrating the same phenomena.

Problem

• Analyze overfitting in Nadaraya-Watson interpolators
• Study generalization across varying hyperparameters
• Identify multiple overfitting regimes in classical methods

Innovation

• Nadaraya-Watson estimator usage
• Bandwidth hyperparameter variation
• Multiple overfitting behaviors analysis