Predictive variational inference: Learn the predictively optimal posterior distribution

2024-10-18

arXiv.org

AI Summary

To address the unreliability of conventional Bayesian posteriors under model misspecification, this paper proposes Predictive Variational Inference (PVI): a paradigm shift from approximating the theoretical posterior to directly optimizing the posterior distribution so that its predictive distribution best approximates the true data-generating process under multiple proper scoring rules. PVI models the posterior as an implicit hierarchical expansion, accommodating both likelihood-based and likelihood-free models, and employs a black-box variational objective without assuming a parametric posterior form. The method inherently enables automated model diagnostics and detection of population-level parameter heterogeneity. Experiments demonstrate that PVI substantially improves predictive accuracy and robustness; moreover, its posterior uncertainty quantitatively exposes model inadequacies and structural heterogeneity, yielding interpretable, diagnosable, and robust inference.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vanilla variational inference finds an optimal approximation to the Bayesian posterior distribution, but even the exact Bayesian posterior is often not meaningful under model misspecification. We propose predictive variational inference (PVI): a general inference framework that seeks and samples from an optimal posterior density such that the resulting posterior predictive distribution is as close to the true data generating process as possible, while this closeness is measured by multiple scoring rules. By optimizing the objective, the predictive variational inference is generally not the same as, or even attempting to approximate, the Bayesian posterior, even asymptotically. Rather, we interpret it as implicit hierarchical expansion. Further, the learned posterior uncertainty detects heterogeneity of parameters among the population, enabling automatic model diagnosis. This framework applies to both likelihood-exact and likelihood-free models. We demonstrate its application in real data examples.

Problem

• Model-Mismatch
• Posterior-Distribution
• Prediction-Accuracy

Innovation

• Predictive Variational Inference
• Hierarchical Modeling
• Model Misspecification Robustness