Beyond State Space Representation: A General Theory for Kernel Packets

2024-02-06

AI Summary

Gaussian process (GP) regression faces scalability bottlenecks in large-scale, high-dimensional settings—existing state-space (SS) models are restricted to 1D regular grids and cannot handle multidimensional scattered data efficiently, while suffering from high computational complexity. Method: This paper introduces Kernel Packing (KP), a general theoretical framework that establishes the first rigorous equivalence between KP and forward–backward state-space representations, enabling exact, approximation-free inference on arbitrary-dimensional scattered data. Contribution/Results: KP achieves linear-time training and logarithmic- or constant-time prediction complexity. By integrating KP construction, forward/backward mappings, and compactly supported covariance compositions, it enables memory-efficient, high-accuracy inference on million-scale additive and multiplicative GP datasets—outperforming both stochastic differential equation (SDE)-based approaches and mainstream low-rank GP methods.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Relational Probabilistic Models
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Gaussian process (GP) regression provides a flexible, nonparametric framework for probabilistic modeling, yet remains computationally demanding in large-scale applications. For one-dimensional data, state space (SS) models achieve linear-time inference by reformulating GPs as stochastic differential equations (SDEs). However, SS approaches are confined to gridded inputs and cannot handle multi-dimensional scattered data. We propose a new framework based on kernel packet (KP), which overcomes these limitations while retaining exactness and scalability. A KP is a compactly supported function defined as a linear combination of the GP covariance functions. In this article, we prove that KPs can be identified via the forward and backward SS representations. We also show that the KP approach enables exact inference with linear-time training and logarithmic or constant-time prediction, and extends naturally to multi-dimensional gridded or scattered data without low-rank approximations. Numerical experiments on large-scale additive and product-form GPs with millions of samples demonstrate that KPs achieve exact, memory-efficient inference where SDE-based and low-rank GP methods fail.

Problem

• Overcoming computational limitations of Gaussian process regression for large datasets
• Extending state space models to handle multi-dimensional scattered data
• Enabling exact linear-time inference without low-rank approximations

Innovation

• Kernel packets enable exact linear-time GP inference
• KP extends state space models to multidimensional data
• KP supports gridded and scattered data without approximations