Fundamental limits of distributed covariance matrix estimation via a conditional strong data processing inequality

2025-07-22

AI Summary

This paper addresses the distributed estimation of high-dimensional covariance matrices under communication constraints in a feature-partitioned setting: multiple agents observe distinct components of a sub-Gaussian random vector and communicate with a central server using finite-bit messages. To characterize fundamental estimation error limits induced by information compression, we introduce the Conditional Strong Data Processing Inequality (C-SDPI), whose coefficient quantifies average information contraction over state-dependent channels—surpassing worst-case analyses and enabling tight, non-asymptotic lower bounds for non-Gaussian settings and finite samples, while revealing communication gains from interactive protocols. Integrating the Geng–Nair tensorization technique, operator Jensen’s inequality, and C-SDPI, we construct a Gaussian-mixture channel model and design a near-optimal estimation protocol. We establish near-tight minimax lower bounds under both operator and Frobenius norms; the proposed protocol achieves matching sample and communication complexities up to logarithmic factors.

Technology Category

• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Machine Learning: Matrix & Tensor Methods
• Intelligent Robots: State Estimation

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Estimating high-dimensional covariance matrices is a key task across many fields. This paper explores the theoretical limits of distributed covariance estimation in a feature-split setting, where communication between agents is constrained. Specifically, we study a scenario in which multiple agents each observe different components of i.i.d. samples drawn from a sub-Gaussian random vector. A central server seeks to estimate the complete covariance matrix using a limited number of bits communicated by each agent. We obtain a nearly tight minimax lower bound for covariance matrix estimation under operator norm and Frobenius norm. Our main technical tool is a novel generalization of the strong data processing inequality (SDPI), termed the Conditional Strong Data Processing Inequality (C-SDPI) coefficient, introduced in this work. The C-SDPI coefficient shares key properties such as tensorization with the conventional SDPI. Crucially, it quantifies the average contraction in a state-dependent channel and can be significantly lower than the worst-case SDPI coefficient over the state input. Utilizing the doubling trick of Geng-Nair and an operator Jensen inequality, we compute this coefficient for Gaussian mixture channels. We then employ it to establish minimax lower bounds on estimation error, capturing the trade-offs among sample size, communication cost, and data dimensionality. Building on this, we present a nearly optimal estimation protocol whose sample and communication requirements match the lower bounds up to logarithmic factors. Unlike much of the existing literature, our framework does not assume infinite samples or Gaussian distributions, making it broadly applicable. Finally, we extend our analysis to interactive protocols, showing interaction can significantly reduce communication requirements compared to non-interactive schemes.

Problem

• Estimate covariance matrices with constrained communication.
• Develop minimax bounds for distributed covariance estimation.
• Analyze trade-offs in sample size, communication, and dimensionality.

Innovation

• Introduces Conditional Strong Data Processing Inequality
• Uses Gaussian mixture channels for contraction analysis
• Develops nearly optimal distributed estimation protocol