Revisiting Penalized Likelihood Estimation for Gaussian Processes

2025-11-22

AI Summary

Gaussian processes (GPs) often suffer from numerical instability in maximum likelihood estimation (MLE) of covariance parameters when modeling expensive simulations with limited data. To address this, we propose a novel cross-validation (CV) metric—Decorrelated Prediction Error (DPE)—which whitens prediction residuals using a Mahalanobis-distance-inspired transformation, thereby mitigating bias and variance inherent in conventional CV errors under small-sample regimes. DPE is embedded within a penalized likelihood framework, automatically selecting an appropriate regularization strength without requiring additional hyperparameter tuning. When regularization is beneficial, DPE significantly outperforms standard CV and the one-standard-error rule; when unnecessary, it matches MLE performance. Extensive experiments demonstrate that DPE offers robustness, broad applicability across GP kernels and designs, and computational feasibility. Overall, DPE establishes a more reliable paradigm for covariance parameter estimation in small-sample GP modeling.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Gaussian processes (GPs) are popular as nonlinear regression models for expensive computer simulations, yet GP performance relies heavily on estimation of unknown covariance parameters. Maximum likelihood estimation (MLE) is common, but it can be plagued by numerical issues in small data settings. The addition of a nugget helps but is not a cure-all. Penalized likelihood methods may improve upon traditional MLE, but their success depends on tuning parameter selection. We introduce a new cross-validation (CV) metric called ``decorrelated prediction error'' (DPE), within the penalized likelihood framework for GPs. Inspired by the Mahalanobis distance, DPE provides more consistent and reliable tuning parameter selection than traditional metrics like prediction error, particularly for $K$-fold CV. Our proposed metric performs comparably to standard MLE when penalization is unnecessary and outperforms traditional tuning parameter selection metrics in scenarios where regularization is beneficial, especially under the one-standard error rule.

Problem

• Improving Gaussian process parameter estimation via penalized likelihood methods
• Addressing unreliable tuning parameter selection in penalized likelihood estimation
• Enhancing cross-validation for regularization parameter selection in GP models

Innovation

• Introducing decorrelated prediction error for tuning
• Improving penalized likelihood estimation in Gaussian processes
• Enhancing cross-validation with Mahalanobis-inspired metric