Scalable Gaussian Processes with Latent Kronecker Structure

2025-06-07

AI Summary

Gaussian processes (GPs) suffer from computational intractability on large-scale, non-gridded, and incompletely observed data—such as real-world time series—where standard Kronecker-based methods fail due to their strict grid-structured assumption. To address this, we propose Latent Kronecker Structure (LKS), a novel framework that models the observed kernel matrix as a linear projection of a latent Kronecker product. LKS enables exact GP inference under arbitrary missingness patterns, breaking the long-standing grid constraint for the first time. Our approach integrates latent Kronecker decomposition, iterative linear solvers, and pathwise conditional sampling—requiring neither sparse approximations nor variational assumptions. Evaluated on real-world tasks in robotics, AutoML, and climate science—with datasets up to 5 million observations—LKS consistently outperforms state-of-the-art sparse and variational GP methods, achieving substantial reductions in computational cost while preserving inference accuracy.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Relational Probabilistic Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Applying Gaussian processes (GPs) to very large datasets remains a challenge due to limited computational scalability. Matrix structures, such as the Kronecker product, can accelerate operations significantly, but their application commonly entails approximations or unrealistic assumptions. In particular, the most common path to creating a Kronecker-structured kernel matrix is by evaluating a product kernel on gridded inputs that can be expressed as a Cartesian product. However, this structure is lost if any observation is missing, breaking the Cartesian product structure, which frequently occurs in real-world data such as time series. To address this limitation, we propose leveraging latent Kronecker structure, by expressing the kernel matrix of observed values as the projection of a latent Kronecker product. In combination with iterative linear system solvers and pathwise conditioning, our method facilitates inference of exact GPs while requiring substantially fewer computational resources than standard iterative methods. We demonstrate that our method outperforms state-of-the-art sparse and variational GPs on real-world datasets with up to five million examples, including robotics, automated machine learning, and climate applications.

Problem

• Addressing computational scalability in large-scale Gaussian Processes
• Overcoming limitations of Kronecker product structures with missing data
• Enabling exact GP inference with reduced computational resources

Innovation

• Leverages latent Kronecker structure for scalability
• Uses iterative linear system solvers efficiently
• Applies pathwise conditioning for exact GPs