Expectations in Expectation Propagation

2025-12-08

AI Summary

Expectation Propagation (EP) suffers from numerical instability and convergence failure under Gaussian projection due to the emergence of negative-variance messages, which cause divergent integrals. Method: Within the linear model framework, this work first systematically characterizes the structural dependencies among EP messages; based on this analysis, it proposes two proactive mitigation strategies—“non-persistent” and “persistent”—that intrinsically prevent the generation of ill-posed, infinitely valued messages. Unlike heuristic regularization approaches, our method relies solely on message propagation path analysis and introduces no additional hyperparameters or approximate corrections. Contribution/Results: Experiments demonstrate that the proposed strategies substantially improve EP’s robustness and convergence speed, effectively suppressing negative-variance messages in both Bayesian linear regression and generalized linear models. This work provides both theoretical insight into EP’s instability mechanisms and a practical, parameter-free tool for stable EP deployment.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Expectation Propagation (EP) is a widely used message-passing algorithm that decomposes a global inference problem into multiple local ones. It approximates marginal distributions (beliefs) using intermediate functions (messages). While beliefs must be proper probability distributions that integrate to one, messages may have infinite integral values. In Gaussian-projected EP, such messages take a Gaussian form and appear as if they have "negative" variances. Although allowed within the EP framework, these negative-variance messages can impede algorithmic progress. In this paper, we investigate EP in linear models and analyze the relationship between the corresponding beliefs. Based on the analysis, we propose both non-persistent and persistent approaches that prevent the algorithm from being blocked by messages with infinite integral values. Furthermore, by examining the relationship between the EP messages in linear models, we develop an additional approach that avoids the occurrence of messages with infinite integral values.

Problem

• Addresses negative-variance messages in Expectation Propagation
• Prevents algorithmic blockage from infinite-integral messages
• Develops approaches to avoid infinite-integral messages in linear models

Innovation

• Prevents algorithm blockage by infinite integral messages
• Develops approach avoiding infinite integral messages occurrence
• Analyzes belief relationships in linear models for solutions