Information Geometry of Message Passing

2026-08-16

AI Summary

This study addresses the accuracy limitations of variational inference in non-conjugate and persistent uncertainty settings by proposing the Natural Gradient Message Passing (NGMP) algorithm. Grounded in information geometry and Forney-style factor graphs, NGMP reformulates stationarity conditions into edge-local forms and employs natural gradient projections to preserve exact messages representable within the receiving family, thereby effectively mitigating precision loss caused by averaging in traditional methods. Empirical evaluations across Poisson smoothing, heteroscedastic regression, and ETTH forecasting tasks demonstrate that NGMP significantly enhances both uncertainty calibration and inference performance in non-conjugate models. These results establish NGMP as a robust approach for improving variational inference accuracy where standard mean-field approximations typically fail due to structural mismatches or sustained uncertainty propagation.

Abstract

We show that the natural-gradient stationary condition of variational inference has an edge-local form on a Forney-style factor graph. We start from the Bethe free energy and constrain a selected edge marginal to an exponential family. At a stationary point, the natural parameter of that edge equals the sum of two projected messages, one from each incident factor. Each projected message is the natural-gradient projection of the exact belief-propagation log-message at the current receiving marginal, or equivalently, the gradient of its expectation in the so-called mean coordinates. We call the resulting scheme natural-gradient message passing (NGMP). The rule is local; each edge may carry its own exponential family, and the message a factor sends depends on the marginal that receives it. Compared with variational message passing, NGMP keeps the part of the exact message that the receiving family can represent instead of averaging the factor under the neighboring beliefs. The two coincide when the uncertainty on the edges entering a non-conjugate factor vanishes, and NGMP is more accurate when that uncertainty persists, for example, along a partially observed latent chain or when parameters are filtered through successive data batches. Experiments on Poisson smoothing, heteroskedastic regression, and hourly ETTh forecasting confirm this and show that the gain appears mainly in uncertainty calibration.

Problem

• Variational Inference
• Message Passing
• Uncertainty Calibration
• Non-conjugate Factors

Innovation

• Natural-gradient message passing
• Information geometry
• Bethe free energy
• Exponential family projection
• Uncertainty calibration