Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions

2025-05-23

AI Summary

This work investigates the training dynamics and fundamental learning limits of Restricted Boltzmann Machines (RBMs) with high-dimensional inputs and a fixed number of hidden units. Addressing the limitation that existing theoretical analyses apply only to special cases reducible to singular value decomposition, the paper establishes a rigorous equivalence between RBM training and a class of multi-index statistical models with non-separable regularization. In the asymptotic limit of diverging input dimension, the analysis integrates Approximate Message Passing (AMP) with its state evolution, the dynamical mean-field theory of gradient descent (GD), and the spiked covariance model to characterize the asymptotic behavior of both AMP and GD. The results demonstrate that RBMs achieve the computational optimal threshold for unsupervised weak recovery—precisely matching the Baik–Ben Arous–Péché (BBP) phase transition point—thereby providing the first rigorous proof of simultaneous achievability of both the information-theoretic limit and the algorithmic performance bound.

Technology Category

• Machine Learning: Learning Theory
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The Restricted Boltzmann Machine (RBM) is one of the simplest generative neural networks capable of learning input distributions. Despite its simplicity, the analysis of its performance in learning from the training data is only well understood in cases that essentially reduce to singular value decomposition of the data. Here, we consider the limit of a large dimension of the input space and a constant number of hidden units. In this limit, we simplify the standard RBM training objective into a form that is equivalent to the multi-index model with non-separable regularization. This opens a path to analyze training of the RBM using methods that are established for multi-index models, such as Approximate Message Passing (AMP) and its state evolution, and the analysis of Gradient Descent (GD) via the dynamical mean-field theory. We then give rigorous asymptotics of the training dynamics of RBM on data generated by the spiked covariance model as a prototype of a structure suitable for unsupervised learning. We show in particular that RBM reaches the optimal computational weak recovery threshold, aligning with the BBP transition, in the spiked covariance model.

Problem

• Analyzing RBM performance in high-dimensional input space
• Simplifying RBM training objective for multi-index model equivalence
• Demonstrating RBM's optimal weak recovery in spiked covariance model

Innovation

• Simplifies RBM training to multi-index model
• Uses AMP and GD for rigorous analysis
• Achieves optimal weak recovery threshold