Tractable Representation Learning with Probabilistic Circuits

2025-07-06

AI Summary

Existing probabilistic circuits (PCs) lack explicit modeling capabilities for probabilistic embeddings, hindering end-to-end representation learning under missing data. To address this, we propose Autoencoding Probabilistic Circuits (APCs), the first framework to enable differentiable and tractable probabilistic embedding learning with PCs. APCs jointly model the observed data distribution and latent variables, enabling native support for arbitrary missingness patterns. A novel differentiable sampling mechanism bridges PCs with neural decoders, yielding an end-to-end trainable hybrid architecture. Experiments demonstrate that APCs significantly outperform PC-based baselines in reconstruction accuracy, achieve embedding quality competitive with state-of-the-art neural models, and exhibit superior robustness under high missingness rates. By unifying tractable probabilistic modeling with deep representation learning, APCs establish a new paradigm for probabilistic representation learning.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Probabilistic circuits (PCs) are powerful probabilistic models that enable exact and tractable inference, making them highly suitable for probabilistic reasoning and inference tasks. While dominant in neural networks, representation learning with PCs remains underexplored, with prior approaches relying on external neural embeddings or activation-based encodings. To address this gap, we introduce autoencoding probabilistic circuits (APCs), a novel framework leveraging the tractability of PCs to model probabilistic embeddings explicitly. APCs extend PCs by jointly modeling data and embeddings, obtaining embedding representations through tractable probabilistic inference. The PC encoder allows the framework to natively handle arbitrary missing data and is seamlessly integrated with a neural decoder in a hybrid, end-to-end trainable architecture enabled by differentiable sampling. Our empirical evaluation demonstrates that APCs outperform existing PC-based autoencoding methods in reconstruction quality, generate embeddings competitive with, and exhibit superior robustness in handling missing data compared to neural autoencoders. These results highlight APCs as a powerful and flexible representation learning method that exploits the probabilistic inference capabilities of PCs, showing promising directions for robust inference, out-of-distribution detection, and knowledge distillation.

Problem

• Enabling tractable probabilistic representation learning with PCs
• Addressing under-explored representation learning in probabilistic circuits
• Handling missing data robustly in autoencoding frameworks

Innovation

• Autoencoding probabilistic circuits for explicit embeddings
• Joint modeling of data and embeddings via PCs
• Hybrid trainable architecture with neural decoder