Influence of Prior Distributions on Gaussian Process Hyperparameter Inference

2025-11-14

AI Summary

Gaussian processes (GPs) in surrogate modeling are highly sensitive to misspecification of covariance hyperparameters—particularly the length-scale parameter θ. While fully Bayesian hierarchical inference improves robustness and uncertainty quantification, its performance critically depends on the choice of prior distributions and Markov Chain Monte Carlo (MCMC) proposal mechanisms—a dependency lacking systematic evaluation in prior work. This paper conducts the first comprehensive study of how alternative priors for θ (uniform, Gamma, inverse-Gamma) and their corresponding MCMC proposals affect posterior sampling efficiency, convergence speed, and predictive performance. Leveraging both synthetic and real-world benchmarks under Bayesian GP inference, we demonstrate that principled alignment between prior and proposal distributions significantly enhances prediction accuracy, improves uncertainty calibration, and accelerates MCMC convergence. Our empirical findings provide actionable guidelines and practical design principles for hyperparameter prior selection in Bayesian GP modeling.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Gaussian processes (GPs) are widely used metamodels for approximating expensive computer simulations, particularly in engineering design and spatial prediction. However, their performance can deteriorate significantly when covariance parameters are poorly estimated, highlighting the importance of accurate inference. The most common approach involves maximizing the marginal likelihood, yielding point estimates of these parameters. However, this approach is highly sensitive to initialization and optimization settings. An alternative is to adopt a fully Bayesian hierarchical framework, where the posterior distribution over the covariance parameters is inferred. This approach provides more robust uncertainty quantification and reduces sensitivity to parameter selection. Yet, a key challenge lies in the careful specification of prior distributions for these parameters. While many available software packages provide default priors, their influence on model behavior is often underexplored. Additionally, the choice of proposal distributions can also influence sampling efficiency and convergence. In this paper, we examine how different prior and proposal distributions over the lengthscale parameters $ heta$ affect predictive performance in a hierarchical GP model, using both simulated and real data experiments. By evaluating various types of priors and proposals, we aim to better understand their influence on predictive accuracy and uncertainty quantification.

Problem

• Investigates how prior distributions affect Gaussian process hyperparameter inference accuracy
• Examines proposal distribution impact on sampling efficiency and model convergence
• Evaluates prior influence on predictive performance and uncertainty quantification

Innovation

• Bayesian hierarchical framework for covariance parameters
• Evaluating prior distributions impact on model performance
• Analyzing proposal distributions for sampling efficiency