Quickest Change Detection for Multiple Data Streams Using the James-Stein Estimator

2024-04-08

arXiv.org

AI Summary

This paper addresses the problem of rapid change-point detection for unknown mean shifts in multiple independent Gaussian data streams. To overcome the excessive detection delay of the conventional MLE-CuSum method in high-dimensional, multi-stream settings, we propose a novel sequential detection method that integrates the James–Stein shrinkage estimator with a window-limited CuSum test. To our knowledge, this is the first approach to embed shrinkage estimation into a sequential testing framework, uniformly reducing detection delay across all post-change parameters under a prescribed false alarm rate constraint. We establish theoretical guarantees showing that, when the number of streams is at least three, the proposed method achieves strictly superior second-order asymptotic detection delay compared to the MLE-CuSum baseline, while preserving minimax optimality. Extensive simulations confirm its substantial reduction in average detection delay for large-scale streaming data, demonstrating both theoretical rigor and practical efficacy.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Data Mining & Knowledge Management: Data Stream Mining
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

The problem of quickest change detection is studied in the context of detecting an arbitrary unknown mean-shift in multiple independent Gaussian data streams. The James-Stein estimator is used in constructing detection schemes that exhibit strong detection performance both asymptotically and non-asymptotically. Our results indicate that utilizing the James-Stein estimator in the recently developed window-limited CuSum test constitutes a uniform improvement over its typical maximum likelihood variant. That is, the proposed James-Stein version achieves a smaller detection delay simultaneously for all possible post-change parameter values and every false alarm rate constraint, as long as the number of parallel data streams is greater than three. Additionally, an alternative detection procedure that utilizes the James-Stein estimator is shown to have asymptotic detection delay properties that compare favorably to existing tests. The second-order asymptotic detection delay term is reduced in a predefined low-dimensional subspace of the parameter space, while second-order asymptotic minimaxity is preserved. The results are verified in simulations, where the proposed schemes are shown to achieve smaller detection delays compared to existing alternatives, especially when the number of data streams is large.

Problem

• Detects unknown mean-shifts in multiple Gaussian data streams
• Improves detection delay using James-Stein estimator
• Enhances performance for large number of data streams

Innovation

• Uses James-Stein estimator for change detection
• Improves window-limited CuSum test performance
• Reduces detection delay in multiple streams