Ridge Boosting is Both Robust and Efficient

2025-10-24

AI Summary

In statistics and machine learning, estimators often face a trade-off between efficiency (low variance) and distributional robustness (low bias across target distributions). This paper proposes Ridge Boosting: a single-step kernel ridge regression augmentation applied to an initial predictor, achieving both robustness to distribution shift and semiparametric efficiency. Crucially, it avoids retraining for each target distribution, unifying robust and efficient estimation within a single theoretical framework. Under the assumption that the nuisance function lies in the unit ball of a reproducing kernel Hilbert space (RKHS), we prove that this one-step augmentation simultaneously attains low bias and optimal asymptotic variance—reaching the semiparametric efficiency bound. Experiments on synthetic data and a real-world task—estimating retirement income age distributions—demonstrate that a single model trained only on source data yields robust and efficient estimates across multiple target quantities.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Intelligent Robots: State Estimation
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Estimators in statistics and machine learning must typically trade off between efficiency, having low variance for a fixed target, and distributional robustness, such as extit{multiaccuracy}, or having low bias over a range of possible targets. In this paper, we consider a simple estimator, emph{ridge boosting}: starting with any initial predictor, perform a single boosting step with (kernel) ridge regression. Surprisingly, we show that ridge boosting simultaneously achieves both efficiency and distributional robustness: for target distribution shifts that lie within an RKHS unit ball, this estimator maintains low bias across all such shifts and has variance at the semiparametric efficiency bound for each target. In addition to bridging otherwise distinct research areas, this result has immediate practical value. Since ridge boosting uses only data from the source distribution, researchers can train a single model to obtain both robust and efficient estimates for multiple target estimands at the same time, eliminating the need to fit separate semiparametric efficient estimators for each target. We assess this approach through simulations and an application estimating the age profile of retirement income.

Problem

• Ridge boosting achieves both efficiency and distributional robustness simultaneously
• It maintains low bias across RKHS unit ball distribution shifts
• Eliminates need for separate semiparametric efficient estimators per target

Innovation

• Ridge boosting combines ridge regression with boosting
• Achieves both efficiency and distributional robustness simultaneously
• Uses only source data for multiple target estimates