Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference

2024-11-01

Neural Information Processing Systems

AI Summary

Gaussian processes (GPs) face a dual challenge in large-scale model selection: prohibitive computational complexity—cubic time and quadratic memory in dataset size $N$—and the difficulty of preserving rigorous uncertainty quantification under approximation. This paper proposes a computation-aware GP model selection framework that explicitly models computational uncertainty and incorporates it into the hyperparameter optimization objective via a novel uncertainty-calibrated loss function. Integrating linear-time hyperparameter optimization with scalable approximate inference, our method achieves $O(N)$ time complexity while strictly maintaining calibrated predictive uncertainty. On a benchmark with 1.8 million data points, our approach completes end-to-end model selection within hours on a single GPU, outperforming state-of-the-art scalable GPs—including SGPR, CGGP, and SVGP—in both accuracy and uncertainty calibration. To the best of our knowledge, this is the first solution for large-scale GP deployment that simultaneously ensures theoretical rigor in uncertainty quantification and practical engineering feasibility.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Model selection in Gaussian processes scales prohibitively with the size of the training dataset, both in time and memory. While many approximations exist, all incur inevitable approximation error. Recent work accounts for this error in the form of computational uncertainty, which enables -- at the cost of quadratic complexity -- an explicit tradeoff between computation and precision. Here we extend this development to model selection, which requires significant enhancements to the existing approach, including linear-time scaling in the size of the dataset. We propose a novel training loss for hyperparameter optimization and demonstrate empirically that the resulting method can outperform SGPR, CGGP and SVGP, state-of-the-art methods for GP model selection, on medium to large-scale datasets. Our experiments show that model selection for computation-aware GPs trained on 1.8 million data points can be done within a few hours on a single GPU. As a result of this work, Gaussian processes can be trained on large-scale datasets without significantly compromising their ability to quantify uncertainty -- a fundamental prerequisite for optimal decision-making.

Problem

• Addresses scalability issues in Gaussian process model selection
• Reduces time and memory costs to linear complexity
• Maintains uncertainty quantification for large datasets

Innovation

• Linear-time scaling for large datasets
• Novel training loss for hyperparameter optimization
• Computational uncertainty tradeoff for precision