Learning from Noise: Effective-Rank Collapse and Out-of-Distribution Rejection in Restricted Boltzmann Machines

2026-07-11

AI Summary

This work addresses the challenge that Restricted Boltzmann Machines (RBMs) often fail to effectively reject out-of-distribution (OOD) samples, frequently misclassifying them as known categories. To mitigate this issue, the authors propose introducing random binary noise during training as an auxiliary rejection class. This strategy induces an effective rank collapse in the visible-layer interaction matrix, concentrating its spectral structure along dominant eigendirections and thereby reshaping the energy landscape to enhance OOD detection capability. The approach integrates RBM energy-based modeling, spectral analysis of the interaction matrix, and Marchenko–Pastur theory from random matrix theory. Empirically, it achieves significantly improved rejection performance on multiple structured OOD datasets while preserving high classification accuracy on MNIST.

Abstract

Restricted Boltzmann machines (RBMs) represent data by shaping an energy landscape over visible and hidden configurations, but their discriminative use is fragile under out-of-distribution (OOD) inputs: samples outside the training distribution can be absorbed into one of the learned class basins rather than rejected. Here, we analyze this failure mode through the spectrum of the induced visible--visible interaction $J=WW^{T}$, where \(W\) is the visible--hidden weight matrix. Relative to a Marchenko--Pastur random-matrix reference, conventional training spreads spectral weight into many weak, bulk-compatible directions, increasing the effective rank of $J$. When auxiliary random binary images are assigned to a rejection label during training, the learned interaction undergoes effective-rank collapse: weak bulk-like modes are depleted, spectral weight concentrates into fewer dominant eigendirections, and the effective rank of $J$ approaches that of the empirical data covariance matrix. The resulting RBM rejects structured OOD image datasets while preserving MNIST classification accuracy, showing that random auxiliary exposure can reshape both the interaction spectrum and the free-energy landscape of an energy-based classifier.

Problem

• out-of-distribution rejection
• Restricted Boltzmann Machines
• effective-rank collapse
• energy-based classification
• OOD detection

Innovation

• effective-rank collapse
• out-of-distribution rejection
• Restricted Boltzmann Machines
• energy-based classification
• spectral shaping