Sharp Phase Transitions in Estimation with Low-Degree Polynomials

2025-02-20

AI Summary

This work bridges the gap between statistical and computational feasibility in high-dimensional planted problems. Focusing on canonical tasks—including planted submatrix detection, dense subgraph recovery, spiked Wigner models, and stochastic block models—it extends the low-degree polynomial method systematically to *exact estimation* (beyond hypothesis testing) for the first time. The analysis rigorously characterizes fundamental thresholds: the Baik–Ben Arous–Péché (BBP) transition and the Kesten–Stigum threshold. Leveraging moment methods, tensor expansions, and information-theoretic lower bounds, the paper establishes a unified hardness framework for $n^delta$-degree polynomials ($delta > 0$), determining optimal polynomial degree thresholds across multiple models. These results provide the first rigorous theoretical validation of the Abbe–Sandon and Lelarge–Miollane conjectures, yielding sharp statistical-computational tradeoffs.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Reasoning under Uncertainty: Graphical Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Security and Privacy: Large-scale security measurements

Abstract

High-dimensional planted problems, such as finding a hidden dense subgraph within a random graph, often exhibit a gap between statistical and computational feasibility. While recovering the hidden structure may be statistically possible, it is conjectured to be computationally intractable in certain parameter regimes. A powerful approach to understanding this hardness involves proving lower bounds on the efficacy of low-degree polynomial algorithms. We introduce new techniques for establishing such lower bounds, leading to novel results across diverse settings: planted submatrix, planted dense subgraph, the spiked Wigner model, and the stochastic block model. Notably, our results address the estimation task -- whereas most prior work is limited to hypothesis testing -- and capture sharp phase transitions such as the"BBP"transition in the spiked Wigner model (named for Baik, Ben Arous, and P'{e}ch'{e}) and the Kesten-Stigum threshold in the stochastic block model. Existing work on estimation either falls short of achieving these sharp thresholds or is limited to polynomials of very low (constant or logarithmic) degree. In contrast, our results rule out estimation with polynomials of degree $n^{delta}$ where $n$ is the dimension and $delta>0$ is a constant, and in some cases we pin down the optimal constant $delta$. Our work resolves open problems posed by Hopkins&Steurer (2017) and Schramm&Wein (2022), and provides rigorous support within the low-degree framework for conjectures by Abbe&Sandon (2018) and Lelarge&Miolane (2019).

Problem

• Sharp phase transitions in estimation
• Lower bounds for low-degree polynomials
• Resolving open problems in high-dimensional estimation

Innovation

• Low-degree polynomial algorithms
• Sharp phase transitions
• High-dimensional estimation tasks