Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits

2024-06-10

Neural Information Processing Systems

AI Summary

Probabilistic integral circuits (PICs) suffer from high memory overhead and poor scalability to complex architectures during large-scale training. Method: We propose a directed acyclic graph (DAG)-structured PIC modeling framework, introducing (i) the first systematic DAG-PIC construction paradigm supporting arbitrary variable decompositions; (ii) a tensorized circuit architecture with neural function sharing to enable efficient differentiable training; and (iii) a systematic numerical integration scheme formalized as a scalable quantized probabilistic circuit (QPC) approximation framework. Contributions/Results: Experiments demonstrate that QPC substantially reduces memory consumption while outperforming conventional probabilistic circuits across multiple benchmarks. Moreover, QPC enhances both the expressive power and scalability of continuous latent variable models, enabling effective learning in high-dimensional and structurally complex settings.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Probabilistic Programming
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Probabilistic integral circuits (PICs) have been recently introduced as probabilistic models enjoying the key ingredient behind expressive generative models: continuous latent variables (LVs). PICs are symbolic computational graphs defining continuous LV models as hierarchies of functions that are summed and multiplied together, or integrated over some LVs. They are tractable if LVs can be analytically integrated out, otherwise they can be approximated by tractable probabilistic circuits (PC) encoding a hierarchical numerical quadrature process, called QPCs. So far, only tree-shaped PICs have been explored, and training them via numerical quadrature requires memory-intensive processing at scale. In this paper, we address these issues, and present: (i) a pipeline for building DAG-shaped PICs out of arbitrary variable decompositions, (ii) a procedure for training PICs using tensorized circuit architectures, and (iii) neural functional sharing techniques to allow scalable training. In extensive experiments, we showcase the effectiveness of functional sharing and the superiority of QPCs over traditional PCs.

Problem

• Scaling continuous latent variable models
• Training DAG-shaped probabilistic integral circuits
• Enhancing scalability with neural functional sharing

Innovation

• DAG-shaped PICs construction
• Tensorized circuit architectures training
• Neural functional sharing techniques