When to Retrain after Drift: A Data-Only Test of Post-Drift Data Size Sufficiency

2026-03-09

AI Summary

This work addresses the challenge of determining when and how much data are sufficient to retrain a model after an abrupt concept drift. To this end, the authors propose CALIPER, a method that leverages a single weighted local regression to analyze the state dependence of post-drift data streams. By jointly examining locality parameters and the monotonic trend of one-step proxy error, CALIPER dynamically assesses whether the current data volume is adequate for stable retraining. Notably, it establishes the first purely data-driven retraining decision mechanism that operates without relying on explicit drift detectors or specific model architectures, thereby bridging the gap between drift detection and effective adaptation. Empirical evaluations across four heterogeneous domains, three learner types, and two drift detectors demonstrate that CALIPER consistently matches or exceeds the performance of optimal fixed-window retraining strategies, with negligible computational overhead and often superior results compared to incremental update approaches.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Models for Web evolution
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Sudden concept drift makes previously trained predictors unreliable, yet deciding when to retrain and what post-drift data size is sufficient is rarely addressed. We propose CALIPER - a detector- and model-agnostic, data-only test that estimates the post-drift data size required for stable retraining. CALIPER exploits state dependence in streams generated by dynamical systems: we run a single-pass weighted local regression over the post-drift window and track a one-step proxy error as a function of a locality parameter $θ$. When an effective sample size gate is satisfied, a monotonically non-increasing trend in this error with increasing a locality parameter indicates that the data size is sufficiently informative for retraining. We also provide a theoretical analysis of our method, and we show that the algorithm has a low per-update time and memory. Across datasets from four heterogeneous domains, three learner families, and two detectors, CALIPER consistently matches or exceeds the best fixed data size for retraining while incurring negligible overhead and often outperforming incremental updates. CALIPER closes the gap between drift detection and data-sufficient adaptation in streaming learning.

Problem

• concept drift
• retraining
• data sufficiency
• streaming learning
• post-drift data

Innovation

• concept drift
• data sufficiency
• streaming learning
• model-agnostic
• local regression