Risk Phase Transitions in Spiked Regression: Alignment Driven Benign and Catastrophic Overfitting

2025-10-01

AI Summary

This paper investigates the generalization error of the minimum-norm interpolating solution in linear regression under a spiked covariance model, focusing on risk phase transitions driven by the interplay among spike strength, dimension-to-sample-size ratio (c = d/n), and alignment between the target function and the leading principal component. Through rigorous derivation of the exact asymptotic expression for the generalization error—analyzed in the (c o infty) regime—we establish, for the first time, a three-phase risk taxonomy: catastrophic, mild, and benign overfitting, governed jointly by these three factors. A key finding is that when both spike strength and target alignment are large, the risk does not monotonically improve; instead, it first deteriorates and then improves—revealing the “double-edged” nature of alignment. The derived phase boundaries precisely characterize critical conditions separating distinct overfitting regimes. Furthermore, we demonstrate that this alignment mechanism retains its qualitative implications—and thus serves as a universal caution—in broader model classes.

Technology Category

• Machine Learning: Optimization
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

This paper analyzes the generalization error of minimum-norm interpolating solutions in linear regression using spiked covariance data models. The paper characterizes how varying spike strengths and target-spike alignments can affect risk, especially in overparameterized settings. The study presents an exact expression for the generalization error, leading to a comprehensive classification of benign, tempered, and catastrophic overfitting regimes based on spike strength, the aspect ratio $c=d/n$ (particularly as $c o infty$), and target alignment. Notably, in well-specified aligned problems, increasing spike strength can surprisingly induce catastrophic overfitting before achieving benign overfitting. The paper also reveals that target-spike alignment is not always advantageous, identifying specific, sometimes counterintuitive, conditions for its benefit or detriment. Alignment with the spike being detrimental is empirically demonstrated to persist in nonlinear models.

Problem

• Characterizing generalization error transitions in overparameterized linear regression
• Identifying conditions for benign versus catastrophic overfitting regimes
• Analyzing counterintuitive effects of target-spike alignment on risk

Innovation

• Analyzes minimum-norm interpolating solutions in regression
• Classifies overfitting regimes using spike strength and alignment
• Reveals alignment can induce detrimental overfitting conditions