Fast Rates for Nonstationary Weighted Risk Minimization

2026-02-05

AI Summary

This work addresses the lack of theoretical guarantees for weighted empirical risk minimization in non-stationary settings by establishing generalization error bounds under distribution drift. By decomposing excess risk into learning and drift errors, the paper develops a unified analysis framework under mixing conditions that applies to arbitrary weight and hypothesis classes. The framework explicitly characterizes the effective sample size induced by the weighting scheme, the complexity of the hypothesis class, and data dependence, thereby relaxing the conventional stationarity assumption. The resulting bounds are valid across diverse settings—including autoregressive processes, linear models, basis function approximation, and neural networks—and recover minimax-optimal convergence rates (up to logarithmic factors) in classical stationary scenarios.

Technology Category

• Machine Learning: Other Foundations of Machine Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Weighted empirical risk minimization is a common approach to prediction under distribution drift. This article studies its out-of-sample prediction error under nonstationarity. We provide a general decomposition of the excess risk into a learning term and an error term associated with distribution drift, and prove oracle inequalities for the learning error under mixing conditions. The learning bound holds uniformly over arbitrary weight classes and accounts for the effective sample size induced by the weight vector, the complexity of the weight and hypothesis classes, and potential data dependence. We illustrate the applicability and sharpness of our results in (auto-) regression problems with linear models, basis approximations, and neural networks, recovering minimax-optimal rates (up to logarithmic factors) when specialized to unweighted and stationary settings.

Problem

• nonstationarity
• weighted risk minimization
• distribution drift
• prediction error
• empirical risk

Innovation

• weighted risk minimization
• nonstationarity
• oracle inequality
• mixing conditions
• effective sample size