drift detection

This work addresses the high false positive rate (12.5%) and false negative rate (6.8%) of Mozilla’s existing T-test–based performance anomaly detection system, which hampers continuous integration efficiency. The authors introduce the first benchmark dataset comprising 174 engineer-annotated performance time series and conduct a systematic evaluation of 25 change-point detection algorithms combined with 15 ensemble strategies. They propose an ensemble voting mechanism that integrates offline, online, and hybrid methods to effectively mitigate the precision–recall trade-off. Experimental results and engineer feedback demonstrate that the proposed approach improves the F1-score by 11% over the original system and has been successfully integrated into Mozilla’s performance engineering infrastructure.

Real-world machine learning systems operate under dynamic data distribution shifts, rendering conventional risk control methods—predicated on static distributional assumptions—ineffective and incapable of online monitoring for decision risk violations. To address this, we propose the first sequential testing framework grounded in the “betting” paradigm, which strictly controls the false alarm rate (≤ α) under arbitrary, unknown distributional shifts—without assuming prior knowledge of drift type or underlying distributions. Our approach unifies betting-based hypothesis testing, risk-bound modeling, and online streaming statistical inference to enable real-time, robust monitoring of model risk. Extensive experiments demonstrate that the method achieves high sensitivity in detecting risk violations across diverse drift scenarios, while simultaneously delivering rigorous statistical guarantees in both anomaly detection and conformal prediction tasks.

Industrial sensor drift causes persistent degradation of machine learning model performance, while standard cross-validation—ignoring temporal dependencies and inducing data leakage—severely overestimates model robustness. To address this, we propose a drift-aware model validation paradigm that eliminates the artificial inflation of drift robustness inherent in conventional validation protocols. Furthermore, we introduce AutoML-DC, the first AutoML system designed for adaptive compensation of diverse sensor drift types, capable of dynamically adjusting to varying drift severities. AutoML-DC integrates data balancing, meta-learning, automated ensemble construction, hyperparameter optimization, feature selection, and gradient boosting into an end-to-end drift compensation learning pipeline. Extensive experiments demonstrate that AutoML-DC significantly outperforms state-of-the-art methods in anomaly detection accuracy, generalizability, and robustness across multiple drift categories, while maintaining consistently high performance under varying drift intensities.

To ensure safety in high-risk AI systems, continuous monitoring for abrupt distributional shifts—such as concept drift, covariate shift, and out-of-support shifts—is essential post-deployment. This paper proposes the Weighted Conformal Test Martingale (WCTM), a generalized nonparametric online changepoint detection framework. WCTM is the first method to achieve *anytime-valid* detection of arbitrary distributional shifts while rigorously controlling the false alarm rate; it further supports online adaptation to mild covariate shifts. Its theoretical foundation integrates weighted conformal prediction, anytime-valid inference, and online martingale construction, unifying the modeling of covariate, concept, and support-set shifts. Evaluated on multiple real-world datasets, WCTM significantly outperforms state-of-the-art methods, achieving superior trade-offs between detection sensitivity and false positive rate, and demonstrating distinct responsiveness to both adaptive and non-adaptive shifts.

Test-time adaptation (TTA) suffers from latent model degradation due to distribution shifts, yet lacks effective online monitoring mechanisms. Method: We propose the first online risk monitoring framework for TTA, leveraging confidence sequences to construct a sequential hypothesis test that dynamically estimates model performance using only unlabeled test samples and detects statistically significant performance deterioration in real time. Contribution/Results: By rigorously integrating statistical inference into TTA—enabling unsupervised, online, and falsifiable failure detection—we bridge a critical methodological gap. Extensive experiments across multiple datasets, diverse shift types (e.g., corruption, domain, semantic), and state-of-the-art TTA algorithms demonstrate that our framework triggers alarms with high precision and low latency, substantially enhancing deployment robustness and safety.

To address the challenge of effectively detecting software aging under dynamic workloads, this paper proposes a machine learning–based detection method incorporating a concept drift adaptation mechanism. For the first time, online concept drift detectors—specifically ADWIN and DDM—are integrated into software aging monitoring, coupled with sliding-window-based tracking of performance metrics to enable real-time responsiveness to abrupt, gradual, and periodic workload transitions. Unlike static models, the proposed approach significantly enhances robustness and generalizability in aging identification for long-running systems. Experimental results demonstrate that the ADWIN-adaptive model achieves a consistently high F1-score above 0.93 across diverse unseen workload scenarios, markedly outperforming conventional methods. This work establishes a scalable, adaptive paradigm for aging-aware early warning in high-reliability systems.

Existing drift detection methods rely on fixed thresholds, struggling to simultaneously minimize false positives and false negatives while exhibiting poor robustness to distributional shifts. This paper proposes a dynamic thresholding mechanism, establishing—through segmented performance analysis and rigorous theoretical proof—that time-varying thresholds are statistically superior to any fixed threshold. The resulting adaptive algorithm requires no prior knowledge and optimizes thresholds online to balance detection sensitivity and stability. Furthermore, a comparative-phase module is introduced to integrate seamlessly with mainstream detectors (e.g., ADWIN, DDM) and extend applicability to multimodal real-world data—including images and tabular datasets. Experiments across 12 synthetic and real-world benchmarks demonstrate that our method reduces false positive rate by 37.2% on average and improves drift recall by 29.5%, significantly enhancing model performance retention under continual learning.

To address the degradation of prediction probability calibration in deployed image classification models due to concept drift, this paper proposes an online calibration monitoring method that requires no access to model internals—only predicted probabilities and ground-truth labels. Our approach introduces, for the first time, a Cumulative Sum (CUSUM) control chart with dynamic control limits into calibration monitoring. It computes cumulative deviations of calibration error over time and adaptively adjusts detection thresholds to enable early warning of calibration loss. Compared to static-threshold methods, our framework significantly enhances sensitivity to temporal distribution shifts and accelerates response to emerging miscalibration. We validate its effectiveness and robustness across multiple image classification benchmarks under diverse concept drift scenarios. The proposed method establishes a scalable, black-box-compatible paradigm for trustworthy model deployment, enabling continuous, lightweight calibration assessment without architectural or training modifications.

To address high temporal uncertainty and difficulty in identifying state transitions in industrial time-series anomaly detection, this paper proposes a synergistic framework integrating piecewise preprocessing with heterogeneous ensemble learning. First, change-point detection algorithms (e.g., ChangeFinder) adaptively segment the time series to explicitly model abrupt system state transitions, thereby mitigating temporal ambiguity. Second, a heterogeneous ensemble model is constructed by fusing Random Forest, XGBoost, and LSTM, augmented with PCA-based feature decorrelation. The method innovatively combines dynamic segmentation with model diversity, establishing a novel paradigm for boundary-sensitive anomaly modeling and feature-weight optimization. Evaluated on real-world industrial datasets, the approach achieves an AUC-ROC of 0.9760—outperforming baseline methods by 11.61 percentage points—and demonstrates significantly enhanced discrimination capability for transient anomalies and concept drift.

Existing comparative evaluations of online regression models across multiple datasets lack rigorous statistical significance testing, particularly under dynamic environments with concept drift. Method: This paper introduces the first principled hypothesis-testing framework for online regression in dynamic settings. It systematically adapts the Friedman test and Nemenyi post-hoc test to multi-dataset online regression evaluation, integrating real and synthetic data, 5-fold cross-validation, and averaging over multiple random seeds to robustly assess convergence stability and adaptation capability under concept drift. Results: Empirical analysis reveals statistically significant performance inconsistencies among mainstream algorithms—including AROW and OGD—highlighting their limited convergence robustness and inadequate responsiveness to concept drift. The work delivers a fully reproducible benchmark framework and empirically grounded insights, establishing a statistical foundation for reliability validation and improvement of online learning algorithms.