Detection and inference of changes in high-dimensional linear regression with non-sparse structures

2024-02-10

AI Summary

This paper addresses multiple change-point detection in high-dimensional linear regression, relaxing the conventional reliance on exact sparsity of regression coefficients or their differences. We propose a novel detection paradigm based on local covariance scanning, which directly characterizes discrepancies in the covariance structure between covariates and responses across subintervals—achieving consistent detection without sparsity assumptions. Theoretically, we establish the first non-asymptotic consistency and statistical inference guarantees under non-sparse, non-Gaussian, temporally dependent, and ultra-high-dimensional settings. Methodologically, we develop a robust inferential framework for post-change-point segment-wise difference parameters and release the open-source R package *inferchange*. Extensive simulations and empirical analysis using macroeconomic data demonstrate that our approach significantly outperforms prevailing sparse methods in both statistical power and robustness.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Non-convex Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

For data segmentation in high-dimensional linear regression settings, the regression parameters are often assumed to be sparse segment-wise, which enables many existing methods to estimate the parameters locally via $ell_1$-regularised maximum likelihood-type estimation and then contrast them for change point detection. Contrary to this common practice, we show that the exact sparsity of neither regression parameters nor their differences, a.k.a. differential parameters, is necessary for consistency in multiple change point detection. In fact, both statistically and computationally, better efficiency is attained by a simple strategy that scans for large discrepancies in local covariance between the regressors and the response. We go a step further and propose a suite of tools for directly inferring about the differential parameters post-segmentation, which are applicable even when the regression parameters themselves are non-sparse. Theoretical investigations are conducted under general conditions permitting non-Gaussianity, temporal dependence and ultra-high dimensionality. Numerical results from simulated and macroeconomic datasets demonstrate the competitiveness and efficacy of the proposed methods. Implementation of all methods is provided in the R package exttt{inferchange} on GitHub.

Problem

• Detects changes in high-dimensional non-sparse regression models
• Proposes tools for inferring differential parameters post-segmentation
• Works under non-Gaussian, temporally dependent, ultra-high dimensional data

Innovation

• Scans local covariance for change detection
• Infers differential parameters post-segmentation
• Handles non-sparse regression parameters effectively