On the Curse of Dimensionality in Private Sparse Covariance Estimation and PCA

2026-06-20

AI Summary

This work addresses the challenge of high-dimensional differentially private covariance estimation and principal component analysis (PCA), which are typically hindered by the curse of dimensionality, requiring sample complexity that scales linearly with the ambient dimension. Under the assumption that the covariance matrix exhibits a $k$-row-column sparsity structure, the authors combine differential privacy mechanisms, sparsity priors, operator norm analysis, and information-theoretic lower bounds to establish—for the first time—an exponential gap in sample complexity between private and non-private settings. Moreover, when the leading eigenvector is also sparse, they propose an algorithm achieving sample complexity dependent only on $\mathrm{poly}(k, \log d)$. The study provides tight upper and lower bounds for differentially private sparse covariance estimation and PCA, significantly advancing the theory of high-dimensional statistics under privacy constraints.

Abstract

We study high-dimensional differentially private (DP) covariance estimation in the operator norm, and principal component analysis (PCA), under $k$-row-column sparsity ($k$-RCS) of the covariance matrix. In the non-private setting, it is known that $\mathsf{poly}(k, \log d)$ samples suffice to solve both of these problems. However, the only comparable result known under DP (Wang et al. 2021) requires $Ω(d)$ samples under standard parameterizations of the problem. We investigate when this curse of dimensionality is inherent for sparse covariance estimation tasks under DP. On the upper bound front, we show that a $\mathsf{poly}(k, \log d)$ sample complexity for PCA is possible under DP, if we also posit sparsity of the leading eigenvector. We complement this result with $\mathsf{poly}(d)$ lower bounds under DP for both sparse covariance estimation and PCA, establishing an exponential gap between the private and non-private variants of these problems when $k = \mathsf{polylog}(d)$. To our knowledge, no such separation has previously been demonstrated for any sparse estimation problems in private high-dimensional statistics. Our techniques are flexible enough that they imply stronger lower bounds even for the well-studied problem of standard DP PCA, without sparsity assumptions.

Problem

• differential privacy
• covariance estimation
• PCA
• sparsity
• curse of dimensionality

Innovation

• differential privacy
• sparse covariance estimation
• PCA
• curse of dimensionality
• sample complexity