Distribution Learning Meets Graph Structure Sampling

2024-05-13

arXiv.org

AI Summary

This paper investigates the intrinsic connection between PAC learning of high-dimensional probabilistic graphical models and graph structure counting/sampling. By linking the average regret of online learning algorithms—specifically Exponentially Weighted Averages (EWA) and Random Walk Metropolis (RWM)—under logarithmic loss to KL-divergence error, it establishes, for the first time, a theoretical bridge between distribution learning and graphical model structure learning. Key contributions include: (1) the first sample-optimal and polynomial-time algorithm for learning Bayesian networks with unknown tree structures; (2) the first polynomial-time algorithm—with optimal sample complexity—for learning Bayesian networks given a chordal skeleton; and (3) novel upper bounds on PAC learning sample complexity, substantially improving upon existing frameworks based on KL-error analysis. Collectively, these results unify three traditionally distinct problems: distribution learning, online prediction, and graphical structure inference.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Graphical Models
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This work establishes a novel link between the problem of PAC-learning high-dimensional graphical models and the task of (efficient) counting and sampling of graph structures, using an online learning framework. We observe that if we apply the exponentially weighted average (EWA) or randomized weighted majority (RWM) forecasters on a sequence of samples from a distribution P using the log loss function, the average regret incurred by the forecaster's predictions can be used to bound the expected KL divergence between P and the predictions. Known regret bounds for EWA and RWM then yield new sample complexity bounds for learning Bayes nets. Moreover, these algorithms can be made computationally efficient for several interesting classes of Bayes nets. Specifically, we give a new sample-optimal and polynomial time learning algorithm with respect to trees of unknown structure and the first polynomial sample and time algorithm for learning with respect to Bayes nets over a given chordal skeleton.

Problem

• Connecting PAC-learning of graphical models with efficient graph structure sampling
• Developing sample-optimal polynomial time algorithms for learning Bayes nets
• Providing first polynomial sample and time algorithm for chordal skeleton Bayes nets

Innovation

• Online learning framework links distribution learning and graph sampling
• EWA and RWM forecasters bound KL divergence via regret analysis
• Polynomial-time algorithms for learning trees and chordal Bayes nets