Accelerated training of Gaussian processes using banded square exponential covariances

2026-01-26

AI Summary

This work addresses the cubic computational complexity associated with inverting the covariance matrix and evaluating its determinant in Gaussian process training. Exploiting the observation that, under the squared exponential kernel, a large proportion of off-diagonal entries in the covariance matrix are nearly zero, the authors propose a banded approximation that preserves the original matrix structure. In the one-dimensional setting, this approach comes with theoretical guarantees on approximation accuracy. By leveraging efficient sparse linear algebra operations, the method substantially reduces the cost of likelihood computation. Compared to variational sparse Gaussian processes, it achieves significantly improved training efficiency while effectively maintaining model performance.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We propose a novel approach to computationally efficient GP training based on the observation that square-exponential (SE) covariance matrices contain several off-diagonal entries extremely close to zero. We construct a principled procedure to eliminate those entries to produce a \emph{banded}-matrix approximation to the original covariance, whose inverse and determinant can be computed at a reduced computational cost, thus contributing to an efficient approximation to the likelihood function. We provide a theoretical analysis of the proposed method to preserve the structure of the original covariance in the 1D setting with SE kernel, and validate its computational efficiency against the variational free energy approach to sparse GPs.

Problem

• Gaussian processes
• square-exponential covariance
• computational efficiency
• covariance matrix
• training acceleration

Innovation

• Gaussian processes
• banded covariance
• square-exponential kernel
• computational efficiency
• likelihood approximation