Deep Gaussian Processes with Gradients

2025-12-19

AI Summary

To address the challenges of ineffective gradient integration and computationally intensive Bayesian inference in deep Gaussian processes (DGPs) for nonstationary computer experiments, this paper proposes Gradient-enhanced Deep Gaussian Processes (Grad-DGP). Grad-DGP employs multi-layer latent Gaussian processes to perform nonlinear input-space warping and gradient-aware modeling. It establishes, for the first time, a complete Bayesian framework supporting gradient embedding—encompassing both gradient-augmented training and gradient posterior prediction. To overcome the O(N³) computational bottleneck, we incorporate an optional Vecchia sparse approximation. Extensive evaluations on multiple nonstationary simulation benchmarks demonstrate that Grad-DGP significantly outperforms both gradient-augmented standard GPs and conventional DGPs, confirming the modeling benefits of incorporating gradient information into deep stochastic mappings. The methodology is publicly available as the R package *deepgp*.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Deep Gaussian processes (DGPs) are popular surrogate models for complex nonstationary computer experiments. DGPs use one or more latent Gaussian processes (GPs) to warp the input space into a plausibly stationary regime, then use typical GP regression on the warped domain. While this composition of GPs is conceptually straightforward, the functional nature of the multi-dimensional latent warping makes Bayesian posterior inference challenging. Traditional GPs with smooth kernels are naturally suited for the integration of gradient information, but the integration of gradients within a DGP presents new challenges and has yet to be explored. We propose a novel and comprehensive Bayesian framework for DGPs with gradients that facilitates both gradient-enhancement and gradient posterior predictive distributions. We provide open-source software in the "deepgp" package on CRAN, with optional Vecchia approximation to circumvent cubic computational bottlenecks. We benchmark our DGPs with gradients on a variety of nonstationary simulations, showing improvement over both GPs with gradients and conventional DGPs.

Problem

• Develops Bayesian framework for deep Gaussian processes with gradients
• Addresses challenges in integrating gradient information within DGPs
• Enhances surrogate modeling for complex nonstationary computer experiments

Innovation

• Deep Gaussian Processes with gradient integration
• Bayesian framework for gradient-enhanced DGP inference
• Vecchia approximation for scalable computational implementation