Functional Estimation of the Marginal Likelihood

2026-02-06

AI Summary

This work addresses the challenge of efficiently estimating and optimizing the marginal likelihood and its functionals in models with high-dimensional parameters and low-dimensional continuous hyperparameters. The authors propose a smooth functional estimation framework based on posterior samples, which integrates Bayesian posterior sampling, function interpolation, and functional estimation techniques. By leveraging posterior samples evaluated at a finite set of grid points, the method constructs a globally smooth estimator of the marginal likelihood. Notably, it unifies sequential Monte Carlo, Gibbs sampling, and Monte Carlo maximum likelihood approaches within a single theoretical framework and establishes estimation consistency under both fixed and dense grid designs. Empirical evaluations on Gaussian process regression and classification, as well as cross-effects models, demonstrate that the approach accurately estimates the marginal likelihood and its associated functionals—including derivatives and integrals.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Probabilistic Circuits and Graphical Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We propose a framework for computing, optimizing and integrating with respect to a smooth marginal likelihood in statistical models that involve high-dimensional parameters/latent variables and continuous low-dimensional hyperparameters. The method requires samples from the posterior distribution of the parameters for different values of the hyperparameters on a simulation grid and returns inference on the marginal likelihood defined everywhere on its domain, and on its functionals. We show how the method relates to many of the methods that have been used in this context, including sequential Monte Carlo, Gibbs sampling, Monte Carlo maximum likelihood, and umbrella sampling. We establish the consistency of the proposed estimators as the sampling effort increases, both when the simulation grid is kept fixed and when it becomes dense in the domain. We showcase the approach on Gaussian process regression and classification and crossed effect models.

Problem

• marginal likelihood
• functional estimation
• high-dimensional parameters
• hyperparameters
• statistical models

Innovation

• marginal likelihood
• functional estimation
• posterior sampling
• hyperparameter inference
• consistency