Testing For Distribution Shifts with Conditional Conformal Test Martingales

2026-02-14

AI Summary

This work addresses the vulnerability of existing distribution shift detection methods to contamination during testing, which often leads to increased detection delay and reduced statistical power. To overcome this limitation, the authors propose a conditional conformal test martingale framework that leverages a fixed reference dataset. By constructing a test martingale that is conditionally valid with respect to the reference data, the method enables sequential detection of arbitrary distribution shifts without suffering from test contamination. The approach provides strict Type I error control at any stopping time, guarantees asymptotic power of one, and ensures bounded expected detection delay. Integrating conditional conformal prediction, test martingale theory, and finite-sample estimation error correction, empirical evaluations demonstrate substantial improvements over standard methods in detection speed, power, and reliability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Distributed Search

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existing CTM detectors construct test martingales by continually growing a reference set with each incoming sample, using it to assess how atypical the new sample is relative to past observations. While this design yields anytime-valid type-I error control, it suffers from test-time contamination: after a change, post-shift observations enter the reference set and dilute the evidence for distribution shift, increasing detection delay and reducing power. In contrast, our method avoids contamination by design by comparing each new sample to a fixed null reference dataset. Our main technical contribution is a robust martingale construction that remains valid conditional on the null reference data, achieved by explicitly accounting for the estimation error in the reference distribution induced by the finite reference set. This yields anytime-valid type-I error control together with guarantees of asymptotic power one and bounded expected detection delay. Empirically, our method detects shifts faster than standard CTMs, providing a powerful and reliable distribution-shift detector.

Problem

• distribution shift
• test-time contamination
• conformal test martingales
• reference set
• detection delay

Innovation

• conditional conformal test martingales
• distribution shift detection
• reference contamination
• anytime-valid inference
• sequential testing