Model-free Change-point Detection using AUC of a Classifier

2024-04-10

AI Summary

Detecting change points in nonstationary, complex time series remains challenging due to the absence of parametric assumptions or low-dimensional structure. Method: We propose a fully model-free offline change-point detection method: (i) partition the sequence into adjacent segments; (ii) train a binary classifier to distinguish samples from consecutive segments; and (iii) use the Area Under the ROC Curve (AUC) as the test statistic, with peaks precisely identifying change points. Contribution/Results: This is the first work to adopt AUC as the core statistic—requiring no distributional assumptions, dimensionality reduction, or prior modeling. We derive the null distribution and establish asymptotic properties under both local and fixed alternatives, proving optimal localization rate. Leveraging sliding-window segmentation, nonparametric testing, and supervised learning, our method significantly outperforms existing model-free approaches on diverse simulations and two real-world datasets, achieving high localization accuracy and strong robustness.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In contemporary data analysis, it is increasingly common to work with non-stationary complex datasets. These datasets typically extend beyond the classical low-dimensional Euclidean space, making it challenging to detect shifts in their distribution without relying on strong structural assumptions. This paper proposes a novel offline change-point detection method that leverages classifiers developed in the statistics and machine learning community. With suitable data splitting, the test statistic is constructed through sequential computation of the Area Under the Curve (AUC) of a classifier, which is trained on data segments on both ends of the sequence. It is shown that the resulting AUC process attains its maxima at the true change-point location, which facilitates the change-point estimation. The proposed method is characterized by its complete nonparametric nature, significant versatility, considerable flexibility, and absence of stringent assumptions on the underlying data or any distributional shifts. Theoretically, we derive the limiting pivotal distribution of the proposed test statistic under null, as well as the asymptotic behaviors under both local and fixed alternatives. The localization rate of the change-point estimator is also provided. Extensive simulation studies and the analysis of two real-world datasets illustrate the superior performance of our approach compared to existing model-free change-point detection methods.

Problem

• Detects distribution shifts in non-stationary complex data
• Uses classifier AUC for model-free change-point estimation
• Avoids strong structural assumptions on data

Innovation

• Uses classifier AUC for change-point detection
• Nonparametric method with no strict assumptions
• Achieves high localization rate for shifts