Optimal Estimation in Orthogonally Invariant Generalized Linear Models: Spectral Initialization and Approximate Message Passing

2026-02-09

AI Summary

This work addresses the challenge that real-world data in generalized linear models often violate the independent and identically distributed (i.i.d.) assumption. Under the relaxed assumption that the design matrix is orthogonally invariant—meaning its singular vectors are uniformly distributed while singular values remain arbitrary—the paper proposes an efficient parameter estimation method combining optimal spectral initialization with Approximate Message Passing (AMP). The proposed approach achieves the information-theoretically optimal sample complexity for weak recovery and attains the fundamental lower bound on estimation error, thereby extending beyond the classical i.i.d. Gaussian design setting. Rigorous theoretical analysis provides strong performance guarantees, and numerical experiments confirm both the algorithm’s effectiveness and the accuracy of the theoretical predictions on orthogonally invariant as well as more general correlated data.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Large-scale security measurements

Abstract

We consider the problem of parameter estimation from a generalized linear model with a random design matrix that is orthogonally invariant in law. Such a model allows the design have an arbitrary distribution of singular values and only assumes that its singular vectors are generic. It is a vast generalization of the i.i.d. Gaussian design typically considered in the theoretical literature, and is motivated by the fact that real data often have a complex correlation structure so that methods relying on i.i.d. assumptions can be highly suboptimal. Building on the paradigm of spectrally-initialized iterative optimization, this paper proposes optimal spectral estimators and combines them with an approximate message passing (AMP) algorithm, establishing rigorous performance guarantees for these two algorithmic steps. Both the spectral initialization and the subsequent AMP meet existing conjectures on the fundamental limits to estimation -- the former on the optimal sample complexity for efficient weak recovery, and the latter on the optimal errors. Numerical experiments suggest the effectiveness of our methods and accuracy of our theory beyond orthogonally invariant data.

Problem

• orthogonally invariant
• generalized linear models
• parameter estimation
• random design matrix
• spectral initialization

Innovation

• orthogonally invariant design
• spectral initialization
• approximate message passing
• optimal estimation
• generalized linear models