Vecchia-Inducing-Points Full-Scale Approximations for Gaussian Processes

2025-07-07

AI Summary

Gaussian processes (GPs) suffer from $O(n^3)$ computational complexity, hindering scalability to large datasets. This paper introduces the Variational Inducing Fields (VIF) framework, the first method to synergistically combine global inducing points with local Vecchia approximations—achieving high accuracy and efficiency across both high- and low-dimensional input spaces. We propose a novel correlation-based neighborhood search strategy, integrated with an enhanced cover tree, to efficiently approximate the residual process. For non-Gaussian likelihoods, we develop an iterative Laplace approximation coupled with a new preconditioner, substantially reducing optimization cost. Implemented in optimized C++ with Python/R interfaces, VIF is integrated into the open-source GPBoost library. Experiments on multiple real-world and synthetic benchmarks demonstrate superior predictive accuracy, enhanced numerical stability, and 10×–1000× speedups in training and prediction over state-of-the-art methods.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Gaussian processes are flexible, probabilistic, non-parametric models widely used in machine learning and statistics. However, their scalability to large data sets is limited by computational constraints. To overcome these challenges, we propose Vecchia-inducing-points full-scale (VIF) approximations combining the strengths of global inducing points and local Vecchia approximations. Vecchia approximations excel in settings with low-dimensional inputs and moderately smooth covariance functions, while inducing point methods are better suited to high-dimensional inputs and smoother covariance functions. Our VIF approach bridges these two regimes by using an efficient correlation-based neighbor-finding strategy for the Vecchia approximation of the residual process, implemented via a modified cover tree algorithm. We further extend our framework to non-Gaussian likelihoods by introducing iterative methods that substantially reduce computational costs for training and prediction by several orders of magnitudes compared to Cholesky-based computations when using a Laplace approximation. In particular, we propose and compare novel preconditioners and provide theoretical convergence results. Extensive numerical experiments on simulated and real-world data sets show that VIF approximations are both computationally efficient as well as more accurate and numerically stable than state-of-the-art alternatives. All methods are implemented in the open source C++ library GPBoost with high-level Python and R interfaces.

Problem

• Scalability issues in Gaussian processes for large datasets
• Bridging global and local approximation methods for efficiency
• Extending framework to non-Gaussian likelihoods with reduced costs

Innovation

• Combines global inducing points and local Vecchia approximations
• Uses correlation-based neighbor-finding strategy
• Extends to non-Gaussian likelihoods with iterative methods