Gaussian Process Tilted Nonparametric Density Estimation using Fisher Divergence Score Matching

2025-04-04

AI Summary

This paper addresses low-dimensional density estimation by proposing the GP-tilted nonparametric density model: a multivariate normal base distribution is flexibly adjusted via an exponentiated Gaussian process (GP) correction term. Methodologically, it introduces the Fisher Variational Predictive Distribution (FVPD)—the first variational inference framework minimizing Fisher divergence—yielding fully closed-form analytical solutions for three canonical Fisher divergence learning problems. By leveraging the equivalence between random Fourier features and single-layer cosine-activated networks, FVPD avoids iterative optimization entirely. Experiments demonstrate high accuracy and training stability across multiple low-dimensional density estimation benchmarks. Crucially, all algorithmic components scale linearly in time complexity, substantially enhancing computational efficiency and scalability for large-scale data.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Learning & Optimization for CV
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Large-scale security measurements

Abstract

We present three Fisher divergence (FD) minimization algorithms for learning Gaussian process (GP) based score models for lower dimensional density estimation problems. The density is formed by multiplying a base multivariate normal distribution with an exponentiated GP refinement, and so we refer to it as a GP-tilted nonparametric density. By representing the GP part of the score as a linear function using the random Fourier feature (RFF) approximation, we show that all learning problems can be solved in closed form. This includes the basic and noise conditional versions of the Fisher divergence, as well as a novel alternative to noise conditional FD models based on variational inference (VI). Here, we propose using an ELBO-like optimization of the approximate posterior with which we derive a Fisher variational predictive distribution. The RFF representation of the GP, which is functionally equivalent to a single layer neural network score model with cosine activation, provides a unique linear form for which all expectations are in closed form. The Gaussian base also helps with tractability of the VI approximation. We demonstrate our three learning algorithms, as well as a MAP baseline algorithm, on several low dimensional density estimation problems. The closed-form nature of the learning problem removes the reliance on iterative algorithms, making this technique particularly well-suited to large data sets.

Problem

• Develop GP-tilted density estimation via Fisher divergence minimization
• Enable closed-form learning with RFF approximation for GP scores
• Propose variational inference alternative for noise conditional models

Innovation

• GP-tilted density with Fisher divergence minimization
• Random Fourier feature for closed-form solutions
• Variational inference for Fisher variational distribution