HiGP: A high-performance Python package for Gaussian Process

2025-03-04

AI Summary

Standard Gaussian processes (GPs) suffer from prohibitive computational complexity and poor scalability on large-scale datasets (thousands to millions of instances). This paper proposes an efficient GP regression and classification framework addressing these limitations. Methodologically, it introduces (1) a novel preconditioning strategy that integrates adaptive Nyström decomposition with customized kernel matrix–vector and matrix–matrix multiplication routines, substantially accelerating convergence and improving numerical stability of iterative solvers such as conjugate gradient; and (2) a PyTorch-based implementation supporting exact gradient computation, multiple kernel acceleration techniques, and end-to-end uncertainty quantification. Empirical results demonstrate that the proposed method achieves 10×–100× speedups in both training and inference over standard GP implementations while preserving modeling accuracy. The framework thus provides a scalable, high-fidelity, and general-purpose solution for large-scale GP applications.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Gaussian Processes (GPs) are flexible, nonparametric Bayesian models widely used for regression and classification tasks due to their ability to capture complex data patterns and provide uncertainty quantification (UQ). Traditional GP implementations often face challenges in scalability and computational efficiency, especially with large datasets. To address these challenges, HiGP, a high-performance Python package, is designed for efficient Gaussian Process regression (GPR) and classification (GPC) across datasets of varying sizes. HiGP combines multiple new iterative methods to enhance the performance and efficiency of GP computations. It implements various effective matrix-vector (MatVec) and matrix-matrix (MatMul) multiplication strategies specifically tailored for kernel matrices. To improve the convergence of iterative methods, HiGP also integrates the recently developed Adaptive Factorized Nystrom (AFN) preconditioner and employs precise formulas for computing the gradients. With a user-friendly Python interface, HiGP seamlessly integrates with PyTorch and other Python packages, allowing easy incorporation into existing machine learning and data analysis workflows.

Problem

• Addresses scalability and computational efficiency in Gaussian Process models.
• Enhances performance of GP regression and classification for large datasets.
• Integrates advanced methods for efficient matrix operations and convergence.

Innovation

• Efficient Gaussian Process regression and classification
• Advanced matrix-vector and matrix-multiplication strategies
• Integration of Adaptive Factorized Nystrom preconditioner