Avoiding Non-Integrable Beliefs in Expectation Propagation

2026-04-05

AI Summary

This work addresses the issue that expectation propagation (EP) may generate non-integrable beliefs during iteration, leading to inference failure—particularly in Bayesian inference tasks such as generalized linear models. To resolve this, the authors propose two novel EP frameworks that enforce integrability of beliefs by constraining the belief space itself rather than imposing restrictive conditions on messages, thereby preserving message flexibility while rigorously guaranteeing belief integrability. These frameworks are derived from a constrained Bethe free energy optimization perspective, seamlessly integrating variational inference with message-passing mechanisms to yield stable iterative update rules. Experimental results demonstrate that the proposed methods effectively avoid non-integrable beliefs in signal recovery tasks under generalized linear models, significantly improving estimation accuracy and extending the applicability of EP to non-standard probabilistic models.

Abstract

Expectation Propagation (EP) is a widely used iterative message-passing algorithm that decomposes a global inference problem into multiple local ones. It approximates marginal distributions as ``beliefs'' using intermediate functions called ``messages''. It has been shown that the stationary points of EP are the same as corresponding constrained Bethe Free Energy (BFE) optimization problem. Therefore, EP is an iterative method of optimizing the constrained BFE. However, the iterative method may fall out of the feasible set of the BFE optimization problem, i.e., the beliefs are not integrable. In most literature, the authors use various methods to keep all the messages integrable. In most Bayesian estimation problems, limiting the messages to be integrable shrinks the actual feasible set. Furthermore, in extreme cases where the factors are not integrable, making the message itself integrable is not enough to have integrable beliefs. In this paper, two EP frameworks are proposed to ensure that EP has integrable beliefs. Both of the methods allows non-integrable messages. We then investigate the signal recovery problem in Generalized Linear Model (GLM) using our proposed methods.

Problem

• Expectation Propagation
• Non-Integrable Beliefs
• Bethe Free Energy
• Message-Passing
• Bayesian Inference

Innovation

• Expectation Propagation
• non-integrable messages
• integrable beliefs
• Bethe Free Energy
• Generalized Linear Model