Gaussian Ensemble Belief Propagation for Efficient Inference in High-Dimensional Systems

2024-02-13

arXiv.org

AI Summary

Bayesian inference in high-dimensional systems faces challenges including prohibitive computational complexity, difficulty integrating black-box models, and failure under strong observational noise. This paper proposes Gaussian Ensemble Belief Propagation (GEnBP), the first algorithm to jointly integrate ensemble Kalman filtering with Gaussian belief propagation within a graphical model framework. GEnBP propagates low-rank local messages along edges to iteratively update a prior ensemble of samples. By operating on compact ensembles, it efficiently handles state spaces whose dimension far exceeds the ensemble size—overcoming fundamental limitations of conventional belief propagation in high dimensions, non-analytic generative processes, and high-noise regimes. Experiments demonstrate that GEnBP significantly outperforms existing BP methods in data assimilation, system identification, and hierarchical modeling: it achieves higher accuracy while reducing computational cost by over an order of magnitude. The method is particularly effective for spatiotemporal dynamical modeling and physics-informed parameter inversion.

Technology Category

• Reasoning under Uncertainty: Probabilistic Inference
• Machine Learning: Probabilistic Circuits and Graphical Models
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Efficient inference in high-dimensional models is a central challenge in machine learning. We introduce the Gaussian Ensemble Belief Propagation (GEnBP) algorithm, which combines the strengths of the Ensemble Kalman Filter (EnKF) and Gaussian Belief Propagation (GaBP) to address this challenge. GEnBP updates ensembles of prior samples into posterior samples by passing low-rank local messages over the edges of a graphical model, enabling efficient handling of high-dimensional states, parameters, and complex, noisy, black-box generation processes. By utilizing local message passing within a graphical model structure, GEnBP effectively manages complex dependency structures and remains computationally efficient even when the ensemble size is much smaller than the inference dimension -- a common scenario in spatiotemporal modeling, image processing, and physical model inversion. We demonstrate that GEnBP can be applied to various problem structures, including data assimilation, system identification, and hierarchical models, and show through experiments that it outperforms existing belief propagation methods in terms of accuracy and computational efficiency. Supporting code is available at https://github.com/danmackinlay/GEnBP

Problem

• Efficient inference in high-dimensional models
• Handling complex, noisy, black-box processes
• Managing complex dependency structures efficiently

Innovation

• Combines EnKF and GaBP
• Low-rank local messages
• Manages complex dependencies efficiently