Computational and statistical lower bounds for low-rank estimation under general inhomogeneous noise

2025-10-09

AI Summary

This paper addresses the detection and estimation of low-rank signals under heterogeneous noise, where noise variances follow an arbitrary (not necessarily block-structured) profile. We develop a novel matrix analysis framework integrating free probability, traffic probability, and graph theory. Our work provides the first rigorous evidence supporting Guionnet et al.’s computational hardness conjecture: when spectral methods fail, no low-degree polynomial algorithm can detect the signal. Information-theoretically, we establish sharp statistical lower bounds. Computationally, we prove that spectral algorithms achieve optimal trade-offs—i.e., they are computationally optimal—for broad classes of signal distributions and smooth variance profiles. Both theoretical analysis and numerical experiments confirm the universality and effectiveness of our approach under general heterogeneous noise, achieving precise alignment between statistical limits and computational feasibility.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Reasoning under Uncertainty: Graphical Models
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Recent work has generalized several results concerning the well-understood spiked Wigner matrix model of a low-rank signal matrix corrupted by additive i.i.d. Gaussian noise to the inhomogeneous case, where the noise has a variance profile. In particular, for the special case where the variance profile has a block structure, a series of results identified an effective spectral algorithm for detecting and estimating the signal, identified the threshold signal strength required for that algorithm to succeed, and proved information-theoretic lower bounds that, for some special signal distributions, match the above threshold. We complement these results by studying the computational optimality of this spectral algorithm. Namely, we show that, for a much broader range of signal distributions, whenever the spectral algorithm cannot detect a low-rank signal, then neither can any low-degree polynomial algorithm. This gives the first evidence for a computational hardness conjecture of Guionnet, Ko, Krzakala, and Zdeborov'a (2023). With similar techniques, we also prove sharp information-theoretic lower bounds for a class of signal distributions not treated by prior work. Unlike all of the above results on inhomogeneous models, our results do not assume that the variance profile has a block structure, and suggest that the same spectral algorithm might remain optimal for quite general profiles. We include a numerical study of this claim for an example of a smoothly-varying rather than piecewise-constant profile. Our proofs involve analyzing the graph sums of a matrix, which also appear in free and traffic probability, but we require new bounds on these quantities that are tighter than existing ones for non-negative matrices, which may be of independent interest.

Problem

• Studying computational optimality of spectral algorithms for low-rank estimation
• Proving sharp information-theoretic lower bounds for signal distributions
• Analyzing low-rank signal detection under general inhomogeneous noise profiles

Innovation

• Spectral algorithm for low-rank signal detection
• Computational hardness via low-degree polynomial analysis
• General variance profiles without block structure assumption