Spectrally Deconfounded Gradient Boosting

2026-07-10

AI Summary

This work addresses the susceptibility of machine learning models to spurious associations induced by hidden confounders and proposes the first spectral debiasing framework tailored for nonlinear gradient boosting. The method replaces the conventional squared error with a spectral loss that suppresses learning rates along confounding directions, integrating early stopping, regularization, and Laplace approximation to achieve effective debiasing. Theoretical analysis elucidates the synergistic mechanism between spectral shrinkage and regularization, establishing connections to LAVA shrinkage and kernel-based random-effects mixed models, which facilitates empirical Bayes hyperparameter tuning. Experiments demonstrate that the proposed approach significantly improves target estimation accuracy on both synthetic and real-world datasets and offers superior scalability compared to existing nonlinear spectral debiasing methods.

Abstract

Flexible machine-learning methods can be sensitive to hidden confounding: they may learn associations induced by unobserved confounders rather than stable signals. Spectral deconfounding mitigates this problem by shrinking high-variance directions of the covariate matrix that, under dense confounding, carry latent confounder information. Existing work has largely focused on linear models. We develop a nonlinear spectral deconfounding framework for gradient boosting. Our approach replaces the ordinary squared-error loss by a spectral loss, which alters the boosting dynamics by slowing down learning in confounding-aligned directions. We show that deconfounding is not achieved by the spectral loss alone, but by the interaction between spectral shrinkage and regularization, especially in terms of early stopping. Moreover, we provide a mixed-model interpretation that connects LAVA-type shrinkage to random-effects adjustment and yields an empirical-Bayes procedure for tuning the spectral loss. We also extend the method to general likelihoods and nonlinear confounding using Laplace approximations and kernel random effects. Across synthetic and real-world experiments, spectrally deconfounded boosting improves estimation of the target function under hidden confounding and is substantially more scalable than existing nonlinear spectral deconfounding baselines.

Problem

• hidden confounding
• gradient boosting
• spectral deconfounding
• nonlinear models
• causal inference

Innovation

• spectral deconfounding
• gradient boosting
• hidden confounding
• regularization
• mixed-effects model