Anytime-Valid Confirmation of Label-Shift Corrections

2026-06-11

AI Summary

This work addresses the challenge of reliably detecting label shift in small-batch scientific deployments, where target-domain labels are scarce. The authors propose a formal sequential testing framework that reframes model monitoring as an anytime-valid sequential hypothesis test. The key innovation lies in constructing a validation rule based on conditional e-values and non-negative martingales, which for the first time directly links the negative log predictive density difference to label shift detection. Theoretical guarantees are established by integrating likelihood ratios, Ville’s inequality, and Gaussian process regression. Simulations demonstrate that the method achieves strong finite-sample power and calibration robustness while rigorously controlling Type I error, significantly outperforming existing approaches that rely on re-estimating label distributions.

Abstract

In small-batch scientific deployments, labeled target outcomes may be too scarce for reliable shift estimation even when unlabeled target inputs are available. We address the complementary setting where the practitioner has a pre-specified label-shift correction from domain knowledge and asks whether incoming labeled outcomes support it. We show that the per-observation likelihood ratio between a label-shift-corrected predictive and the source predictive is a conditional e-value, so its running product is a nonnegative martingale and Ville's inequality yields an anytime-valid confirmation rule. The log martingale equals the cumulative negative log-predictive density (NLPD) gap between the source and the corrected predictive, converting routine model monitoring into a formal sequential test. Rejection means the incoming data support the posited correction relative to the source predictive, but it is not a precise estimate of the degree of shift. Closed forms are available for GP sources with Gaussian label-shift ratios. GP regression simulations validate Type I control, finite-sample power, miscalibration sensitivity, and the small-batch advantage of a reliable prior over label-based re-estimation.

Problem

• label-shift
• anytime-valid inference
• small-batch deployment
• predictive monitoring
• distribution shift

Innovation

• anytime-valid inference
• label-shift correction
• e-values
• martingales
• sequential testing