A Compositional Theory of Curvature in Probabilistic Circuits

2026-08-13

AI Summary

This work addresses the limitations of global sharpness-aware regularization in probabilistic circuits, which induces depth bias and underfitting by disregarding the compositional curvature structure of the loss landscape. The study reveals, for the first time, that the trace of the Hessian in probabilistic circuits admits a decomposable form, precisely factorizing into the product of circuit flows and local sharpness terms. Building on this insight, the authors propose an adaptive sharpness-aware regularization method grounded in local intrinsic curvature. This approach preserves the closed-form updates of the EM algorithm while effectively balancing model generalization and training stability. Empirical results demonstrate that the proposed method substantially recovers the generalization performance sacrificed by global regularization, without compromising the robustness inherent to sharpness-aware learning.

Abstract

Probabilistic Circuits (PCs) are generative models that support exact inference and, unlike deep neural networks, admit an exact and tractable measure of loss-surface curvature: the trace of the Hessian of the log-likelihood. Recent work regularizes this trace globally to bias learning toward flatter, better generalizing optima. We show that treating sharpness as a global regularizer can be misspecified for PCs, whose curvature is inherently compositional. We prove that each sum node's contribution to the Hessian trace factorizes exactly into its circuit flow, which measures how heavily the node is used, and a local sharpness term determined by its output distribution. This decomposition provides insights into why global sharpness regularization is depth biased and can lead to underfitting. Building on it, we introduce an adaptive sharpness aware regularizer that penalizes nodes based on intrinsic local curvature and preserves closed form EM updates. We also show that empirically, this targeted regularization recovers the generalization that global regularization sacrifices while retaining the robustness and benefits of sharpness aware learning.

Problem

• Probabilistic Circuits
• curvature
• sharpness regularization
• compositional structure
• Hessian trace

Innovation

• Probabilistic Circuits
• Hessian trace
• compositional curvature
• sharpness-aware regularization
• local sharpness