A Spectral Framework for Closed-Form Relative Density Estimation

2026-05-11

AI Summary

This work addresses the absence of closed-form solutions for relative log-densities in linearly parameterized probabilistic models—including unnormalized and conditional models—by proposing a closed-form spectral estimator that relies solely on first- and second-order feature moments. By expressing the Kullback–Leibler divergence as an integral of weighted chi-squared divergences, the problem is recast as a family of least-squares problems, yielding explicit spectral formulas applicable to a broad class of f-divergences. These formulas enable direct estimation of both divergences and log-density potentials. The approach naturally integrates with kernel methods or neural networks for feature learning, enjoys theoretical consistency guarantees, and demonstrates substantial empirical advantages over optimization-based variational methods—such as logistic and softmax regression—on synthetic data.

Abstract

We propose a closed-form spectral framework for relative log-density estimation in linearly parameterized probabilistic models, including unnormalized and conditional models. This is achieved by representing the Kullback-Leibler (KL) divergence as an integral of weighted chi-squared divergences, converting KL estimation into a family of least-squares problems. We derive an explicit spectral formula based only on first- and second-order feature moments, yielding closed-form estimators of both divergences and log-density potentials for fixed features. The framework extends to a broad class of f-divergences and can be combined with kernelization or feature learning with neural networks. We prove convergence guarantees for the resulting estimators and empirically compare them on synthetic data with optimization-based variational formulations, including logistic and softmax regression for normalized conditional models.

Problem

• relative density estimation
• linearly parameterized probabilistic models
• KL divergence
• f-divergences
• log-density potentials

Innovation

• spectral estimation
• relative density estimation
• closed-form estimator
• f-divergence
• moment-based method