Empirical Gaussian Processes

2026-02-12

AI Summary

This work proposes an empirical Gaussian process framework that overcomes the limitations of traditional Gaussian processes, which rely on handcrafted kernel functions and struggle to capture complex covariance structures in real-world data. By directly learning both mean and covariance functions from multi-source, heterogeneous historical datasets, the method constructs a data-driven prior that transcends the constraints of parametric kernels. An expectation–maximization (EM) algorithm with closed-form updates is derived via likelihood estimation, effectively addressing the challenge of inconsistent observation locations across datasets and approximating the true data-generating process in terms of KL divergence. Empirical evaluations demonstrate competitive performance in learning curve extrapolation and time series forecasting tasks.

Technology Category

• Machine Learning: Kernel Methods
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Gaussian processes (GPs) are powerful and widely used probabilistic regression models, but their effectiveness in practice is often limited by the choice of kernel function. This kernel function is typically handcrafted from a small set of standard functions, a process that requires expert knowledge, results in limited adaptivity to data, and imposes strong assumptions on the hypothesis space. We study Empirical GPs, a principled framework for constructing flexible, data-driven GP priors that overcome these limitations. Rather than relying on standard parametric kernels, we estimate the mean and covariance functions empirically from a corpus of historical observations, enabling the prior to reflect rich, non-trivial covariance structures present in the data. Theoretically, we show that the resulting model converges to the GP that is closest (in KL-divergence sense) to the real data generating process. Practically, we formulate the problem of learning the GP prior from independent datasets as likelihood estimation and derive an Expectation-Maximization algorithm with closed-form updates, allowing the model handle heterogeneous observation locations across datasets. We demonstrate that Empirical GPs achieve competitive performance on learning curve extrapolation and time series forecasting benchmarks.

Problem

• Gaussian processes
• kernel selection
• data-driven priors
• covariance structure
• prior adaptivity

Innovation

• Empirical Gaussian Processes
• data-driven priors
• nonparametric covariance estimation
• Expectation-Maximization algorithm
• KL-divergence convergence