Scalable Gaussian Processes: Advances in Iterative Methods and Pathwise Conditioning

2025-07-09

AI Summary

Gaussian processes (GPs) suffer from cubic time and quadratic space complexity, hindering scalability to large datasets and modern parallel hardware. To address this, we propose a novel scalable GP inference framework that unifies iterative linear solvers with pathwise conditioning—replacing Cholesky-based inference with matrix-multiplication-dominated iterative linear system solving. This reformulation reduces memory complexity from $O(n^2)$ to $O(n)$, enables efficient GPU/TPU acceleration, and supports training and prediction on datasets with over one million points. Crucially, the method preserves exact GP uncertainty quantification without approximation of the posterior distribution. Empirical evaluation demonstrates that our approach achieves several-fold speedups over state-of-the-art sparse and variational GP methods while delivering superior predictive accuracy and calibrated uncertainty estimates.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Programming
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Gaussian processes are a powerful framework for uncertainty-aware function approximation and sequential decision-making. Unfortunately, their classical formulation does not scale gracefully to large amounts of data and modern hardware for massively-parallel computation, prompting many researchers to develop techniques which improve their scalability. This dissertation focuses on the powerful combination of iterative methods and pathwise conditioning to develop methodological contributions which facilitate the use of Gaussian processes in modern large-scale settings. By combining these two techniques synergistically, expensive computations are expressed as solutions to systems of linear equations and obtained by leveraging iterative linear system solvers. This drastically reduces memory requirements, facilitating application to significantly larger amounts of data, and introduces matrix multiplication as the main computational operation, which is ideal for modern hardware.

Problem

• Improving scalability of Gaussian processes for large data
• Reducing memory usage via iterative linear system solvers
• Optimizing computations for modern parallel hardware

Innovation

• Combines iterative methods with pathwise conditioning
• Uses iterative linear system solvers
• Optimizes matrix multiplication for modern hardware