WATCH: Weighted Adaptive Testing for Changepoint Hypotheses via Weighted-Conformal Martingales

2025-05-07

AI Summary

To ensure safety in high-risk AI systems, continuous monitoring for abrupt distributional shifts—such as concept drift, covariate shift, and out-of-support shifts—is essential post-deployment. This paper proposes the Weighted Conformal Test Martingale (WCTM), a generalized nonparametric online changepoint detection framework. WCTM is the first method to achieve *anytime-valid* detection of arbitrary distributional shifts while rigorously controlling the false alarm rate; it further supports online adaptation to mild covariate shifts. Its theoretical foundation integrates weighted conformal prediction, anytime-valid inference, and online martingale construction, unifying the modeling of covariate, concept, and support-set shifts. Evaluated on multiple real-world datasets, WCTM significantly outperforms state-of-the-art methods, achieving superior trade-offs between detection sensitivity and false positive rate, and demonstrating distinct responsiveness to both adaptive and non-adaptive shifts.

Technology Category

• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Responsibly deploying artificial intelligence (AI) / machine learning (ML) systems in high-stakes settings arguably requires not only proof of system reliability, but moreover continual, post-deployment monitoring to quickly detect and address any unsafe behavior. Statistical methods for nonparametric change-point detection -- especially the tools of conformal test martingales (CTMs) and anytime-valid inference -- offer promising approaches to this monitoring task. However, existing methods are restricted to monitoring limited hypothesis classes or ``alarm criteria,'' such as data shifts that violate certain exchangeability assumptions, or do not allow for online adaptation in response to shifts. In this paper, we expand the scope of these monitoring methods by proposing a weighted generalization of conformal test martingales (WCTMs), which lay a theoretical foundation for online monitoring for any unexpected changepoints in the data distribution while controlling false-alarms. For practical applications, we propose specific WCTM algorithms that accommodate online adaptation to mild covariate shifts (in the marginal input distribution) while raising alarms in response to more severe shifts, such as concept shifts (in the conditional label distribution) or extreme (out-of-support) covariate shifts that cannot be easily adapted to. On real-world datasets, we demonstrate improved performance relative to state-of-the-art baselines.

Problem

• Detecting unexpected changepoints in data distribution
• Controlling false-alarms during online monitoring
• Adapting to mild covariate shifts while alarming severe shifts

Innovation

• Weighted conformal test martingales for changepoint detection
• Online adaptation to mild covariate shifts
• Improved performance on real-world datasets