The Gaussian-Multinoulli Restricted Boltzmann Machine: A Potts Model Extension of the GRBM

2025-05-16

AI Summary

Real-world tasks—such as associative memory and symbolic reasoning—require discrete, structured representations, yet continuous latent variable models struggle to capture such structures naturally. To address this, we propose the Gaussian–multivalued Potts Restricted Boltzmann Machine (GP-RBM), which replaces binary hidden units with q-state Potts variables, thereby inducing an interpretable, multivalued latent concept space. By integrating a Gaussian–categorical joint distribution with an energy-based Potts statistical mechanics framework, GP-RBM ensures tractable inference while enabling efficient training via contrastive divergence. Empirically, GP-RBM significantly improves modeling of multimodal distributions and outperforms conventional binary RBMs on simulated recall and analogical reasoning tasks. These results demonstrate its superior capacity for discrete representation learning, combinatorial expressivity, and scalability—establishing a principled foundation for structured, interpretable latent modeling in energy-based architectures.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Many real-world tasks, from associative memory to symbolic reasoning, demand discrete, structured representations that standard continuous latent models struggle to express naturally. We introduce the Gaussian-Multinoulli Restricted Boltzmann Machine (GM-RBM), a generative energy-based model that extends the Gaussian-Bernoulli RBM (GB-RBM) by replacing binary hidden units with $q$-state Potts variables. This modification enables a combinatorially richer latent space and supports learning over multivalued, interpretable latent concepts. We formally derive GM-RBM's energy function, learning dynamics, and conditional distributions, showing that it preserves tractable inference and training through contrastive divergence. Empirically, we demonstrate that GM-RBMs model complex multimodal distributions more effectively than binary RBMs, outperforming them on tasks involving analogical recall and structured memory. Our results highlight GM-RBMs as a scalable framework for discrete latent inference with enhanced expressiveness and interoperability.

Problem

• Extends GB-RBM with Potts variables for richer discrete representations
• Enables learning multivalued interpretable latent concepts effectively
• Improves modeling complex multimodal distributions over binary RBMs

Innovation

• Extends GB-RBM with q-state Potts variables
• Enables richer discrete latent representations
• Maintains tractable inference via contrastive divergence