Generalization Error Estimation for Primal--Dual Algorithms in Non-Smooth Regression

2026-08-13

AI Summary

This study addresses the challenge of estimating generalization error along algorithmic trajectories in nonsmooth regression. We propose a recursive risk estimation framework based on dual-iteration weighted correction to overcome the failure of traditional gradient-based risk estimators on nonsmooth optimization paths. By constructing a covariance-independent universal estimator and employing an observable derivative contraction mechanism as a surrogate for weighting, our method provides rigorous finite-sample guarantees and Gaussian universality. Empirical results demonstrate that the proposed framework accurately tracks out-of-sample risk along finite optimization trajectories, offering a reliable, data-driven criterion for model selection in nonsmooth optimization settings.

Abstract

This paper studies trajectory-wise estimation of generalization error for primal--dual algorithms in non-smooth regression. Motivating examples include \(\ell_1\)-penalized least absolute deviations regression and square-root Lasso regression, where the data-fitting loss is non-differentiable and existing risk estimators for gradient-type optimization paths do not apply directly. We develop a general recursive framework that includes the Chambolle--Pock algorithm and related primal--dual splitting methods. We estimate risk by correcting each in-sample fitted value with a weighted combination of past dual iterates. The ideal weights are Stein derivative contractions and depend on the design covariance. We construct replacement weights from observable derivative contractions of the fitted-signal trajectory, yielding a covariance-free, data-driven correction. For high-dimensional Gaussian designs and fixed finite iteration horizon, we prove finite-sample guarantees for both estimators. For square-root ridge, we further establish a matched-Gaussian universality result beyond Gaussian designs. Numerical experiments show that the proposed estimators accurately track the out-of-sample risk along finite optimization paths.

Problem

• Generalization Error Estimation
• Primal-Dual Algorithms
• Non-Smooth Regression
• Risk Estimation

Innovation

• Primal-Dual Algorithms
• Non-Smooth Regression
• Generalization Error Estimation
• Covariance-Free Correction
• Stein Derivative Contractions