Universality of first-order methods on random and deterministic matrices

2026-04-13

AI Summary

This study investigates the asymptotic behavior of generalized first-order methods on structured—particularly deterministic—matrices, with the aim of constructing iterative algorithms that preserve conditional Gaussianity. By introducing the theory of limiting traffic distributions together with graph expansion techniques, the authors compute for the first time the traffic distribution of deterministic matrices such as Walsh–Hadamard ensembles. Leveraging this result, they propose a novel approximate message passing (AMP) algorithm that exhibits universal Gaussian dynamics across a broad class of matrices, and provide a combinatorial interpretation of its Onsager correction term. This work unifies and extends existing AMP variants, resolving a conjecture partially posed by Marinari et al. in 1994.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements

Abstract

General first-order methods (GFOM) are a flexible class of iterative algorithms which update a state vector by matrix-vector multiplications and entrywise nonlinearities. A long line of work has sought to understand the large-n dynamics of GFOM, mostly focusing on "very random" input matrices and the approximate message passing (AMP) special case of GFOM whose state is asymptotically Gaussian. Yet, it has long remained unknown how to construct iterative algorithms that retain this Gaussianity for more structured inputs, or why existing AMP algorithms can be as effective for some deterministic matrices as they are for random matrices. We analyze diagrammatic expansions of GFOM via the limiting traffic distribution of the input matrix, the collection of all limiting values of permutation-invariant polynomials in the matrix entries, to obtain the following results: 1. We calculate the traffic distribution for the first non-trivial deterministic matrices, including (minor variants of) the Walsh-Hadamard and discrete sine and cosine transform matrices. This determines the limiting dynamics of GFOM on these inputs, resolving parts of longstanding conjectures of Marinari, Parisi, and Ritort (1994). 2. We design a new AMP iteration which unifies several previous AMP variants and generalizes to new input types, whose limiting dynamics are Gaussian conditional on some latent random variables. The asymptotic dynamics hold for a large and natural class of traffic distributions (encompassing both random and deterministic input matrices) and the algorithm's analysis gives a simple combinatorial interpretation of the Onsager correction, answering questions posed recently by Wang, Zhong, and Fan (2022).

Problem

• first-order methods
• approximate message passing
• deterministic matrices
• random matrices
• traffic distribution

Innovation

• Approximate Message Passing
• Traffic Distribution
• General First-Order Methods
• Deterministic Matrices
• Onsager Correction