Combinatorial Sparse PCA Beyond the Spiked Identity Model

2026-03-03

AI Summary

This work proposes the first combinatorial sparse principal component analysis (Sparse PCA) algorithm applicable to arbitrary covariance matrices, overcoming the limitation of existing methods that are restricted to the spiked identity covariance model. By refining the truncated power iteration approach of Yuan and Zhang (2013) and integrating techniques for sparse eigenvector estimation with global convergence analysis, the method achieves accurate recovery of sparse principal components under general covariance structures. The algorithm requires only \(s^2 \cdot \text{polylog}(d)\) samples and runs in \(d^2 \cdot \text{poly}(s, \log d)\) time, where \(s\) denotes the sparsity level and \(d\) the ambient dimension. Both theoretical guarantees and empirical experiments demonstrate its effectiveness and superiority over prior approaches.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Data Mining & Knowledge Management: Data Compression

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements

Abstract

Sparse PCA is one of the most well-studied problems in high-dimensional statistics. In this problem, we are given samples from a distribution with covariance $Σ$, whose top eigenvector $v \in R^d$ is $s$-sparse. Existing sparse PCA algorithms can be broadly categorized into (1) combinatorial algorithms (e.g., diagonal or elementwise covariance thresholding) and (2) SDP-based algorithms. While combinatorial algorithms are much simpler, they are typically only analyzed under the spiked identity model (where $Σ= I_d + γvv^\top$ for some $γ> 0$), whereas SDP-based algorithms require no additional assumptions on $Σ$. We demonstrate explicit counterexample covariances $Σ$ against the success of standard combinatorial algorithms for sparse PCA, when moving beyond the spiked identity model. In light of this discrepancy, we give the first combinatorial method for sparse PCA that provably succeeds for general $Σ$ using $s^2 \cdot \mathrm{polylog}(d)$ samples and $d^2 \cdot \mathrm{poly}(s, \log(d))$ time, by providing a global convergence guarantee on a variant of the truncated power method of Yuan and Zhang (2013). We provide a natural generalization of our method to recovering a vector in a sparse leading eigenspace. Finally, we evaluate our method on synthetic and real-world sparse PCA datasets.

Problem

• Sparse PCA
• combinatorial algorithms
• spiked identity model
• covariance estimation
• high-dimensional statistics

Innovation

• Combinatorial Sparse PCA
• Spiked Identity Model
• Truncated Power Method
• Global Convergence
• High-dimensional Statistics