Sum of Squares Circuits

2024-08-21

AAAI Conference on Artificial Intelligence

AI Summary

This paper addresses the fundamental trade-off between expressiveness and tractability in probabilistic generative modeling. We propose Sum-of-Squares Probabilistic Circuits (SoS-PCs), a novel class of models that support exact inference in polynomial time while achieving strong expressive power. We formally define SoS-PCs, establish a rigorous expressivity hierarchy, and prove exponential advantages over monotone and squared PCs. SoS-PCs unify several complex-parameter tractable models—including Born machines and PSD models—by integrating sum-of-squares structure over the reals with complex-valued parameterization, guided by algebraic complexity theory and tractability constraints. This design enables synergistic gains in both representational capacity and computational efficiency. Empirical evaluation demonstrates that SoS-PCs significantly outperform existing tractable generative models on distribution estimation tasks.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Designing expressive generative models that support exact and efficient inference is a core question in probabilistic ML. Probabilistic circuits (PCs) offer a framework where this tractability-vs-expressiveness trade-off can be analyzed theoretically. Recently, squared PCs encoding subtractive mixtures via negative parameters have emerged as tractable models that can be exponentially more expressive than monotonic PCs, i.e., PCs with positive parameters only. In this paper we provide a more precise theoretical characterization of the expressiveness relationships among these models. First, we prove that squared PCs can be less expressive than monotonic ones. Second, we formalize a novel class of PCs – sum of squares PCs – that can be exponentially more expressive than both squared and monotonic PCs. Around sum of squares PCs, we build an expressiveness hierarchy that allows us to precisely unify and separate different tractable model classes such as Born Machines and PSD models, and other recently introduced tractable probabilistic models by using complex parameters. Finally, we empirically show the effectiveness of sum of squares circuits in performing distribution estimation.

Problem

• Analyzing expressiveness trade-offs in probabilistic circuits
• Introducing sum of squares PCs for higher expressiveness
• Unifying tractable model classes via expressiveness hierarchy

Innovation

• Sum of Squares PCs enhance expressiveness exponentially
• Negative parameters enable subtractive mixtures in PCs
• Complex parameters unify tractable model classes