Automatic Laplace Collapsed Sampling: Scalable Marginalisation of Latent Parameters via Automatic Differentiation

2026-03-27

AI Summary

This work addresses the computational challenge of marginalizing latent variables in high-dimensional Bayesian models, where conventional approaches rely on manual derivation of gradients and Hessian matrices, limiting scalability. The authors propose the ALCS framework, which integrates MAP optimization with Laplace approximation at each step of nested sampling. By leveraging automatic differentiation, ALCS compresses high-dimensional latent contributions into scalar terms, enabling efficient exploration solely within the low-dimensional hyperparameter space. This approach achieves, for the first time, scalable marginalization without manual analytic derivations and accommodates local approximations such as the Student-t distribution. Additionally, a posterior effective sample size (ESS) diagnostic is introduced to detect regions where the approximation breaks down. Experiments demonstrate that ALCS substantially reduces computational dimensionality and enhances both the efficiency and applicability of Bayesian evidence estimation across hierarchical, time-series, and discrete-likelihood models.

Abstract

We present Automatic Laplace Collapsed Sampling (ALCS), a general framework for marginalising latent parameters in Bayesian models using automatic differentiation, which we combine with nested sampling to explore the hyperparameter space in a robust and efficient manner. At each nested sampling likelihood evaluation, ALCS collapses the high-dimensional latent variables $z$ to a scalar contribution via maximum a posteriori (MAP) optimisation and a Laplace approximation, both computed using autodiff. This reduces the effective dimension from $d_θ+ d_z$ to just $d_θ$, making Bayesian evidence computation tractable for high-dimensional settings without hand-derived gradients or Hessians, and with minimal model-specific engineering. The MAP optimisation and Hessian evaluation are parallelised across live points on GPU-hardware, making the method practical at scale. We also show that automatic differentiation enables local approximations beyond Laplace to parametric families such as the Student-$t$, which improves evidence estimates for heavy-tailed latents. We validate ALCS on a suite of benchmarks spanning hierarchical, time-series, and discrete-likelihood models and establish where the Gaussian approximation holds. This enables a post-hoc ESS diagnostic that localises failures across hyperparameter space without expensive joint sampling.

Problem

• marginalisation
• latent parameters
• Bayesian evidence
• high-dimensional inference
• nested sampling

Innovation

• Automatic Differentiation
• Laplace Approximation
• Nested Sampling
• Marginalisation
• Bayesian Evidence