Regularized Variational and Spectral Log-Density-Ratio Estimation in the Gaussian Location Model

2026-07-02

AI Summary

This study addresses the problem of regularized log-density ratio estimation under a high-dimensional Gaussian location model with ridge regularization. Focusing on the limited-sample regime, the authors construct two estimators—one based on variational methods and the other on spectral techniques—and, for the first time, derive their deterministic equivalents within a high-dimensional asymptotic framework. By integrating the convex Gaussian min-max theorem, resolvent analysis from random matrix theory, and variational characterizations of KL divergence, the theoretical analysis reveals that the variational estimator achieves lower risk in large-sample settings, whereas the spectral estimator is superior in small-sample scenarios due to its reduced variance. Numerical experiments validate the critical roles of optimal regularization strategies and the signal-to-dimension ratio in estimator performance, and further demonstrate that incorporating nuclear norm penalization enhances feature learning capabilities.

Abstract

We study ridge-regularized log-density-ratio estimation in the Gaussian location model with a common covariance matrix. By affine invariance, the model is written as q $\sim$ N(0, I), p $\sim$ N($Δ$, I), with linear features, where $Δ$ is a mean vector. The variational estimator is the empirical Kullback-Leibler (KL) log-normalized fit with a squared L2-penalty on its nonconstant coefficient, and the spectral estimator recently introduced in [1] replaces a single variational problem by a continuum of ridge-regularized least-squares problems. We derive high-dimensional deterministic asymptotic equivalents when the numbers of observations and dimension tend to infinity with fixed ratios. The regularized variational limit is characterized by a scalar entropy minimization problem derived from the convex-Gaussian-min-max theorem (CGMT), while the regularized spectral limit follows from deterministic equivalents for resolvents of weighted sums of two independent Gaussian sample covariance matrices. We use these formulas to compare population risks, with experiments focused on fixed-signal aspect-ratio sweeps and optimized regularization. Our conclusion is that with many observations, under the criteria and asymptotic regimes analyzed here, the well-specified variational estimator has the smaller risk, while with fewer observations, the spectral estimator is favored because its covariance-based construction has lower variance. We also study how a nuclear penalty can be used and partially analyzed to perform feature learning.

Problem

• log-density-ratio estimation
• Gaussian location model
• ridge regularization
• high-dimensional asymptotics
• feature learning

Innovation

• log-density-ratio estimation
• ridge regularization
• high-dimensional asymptotics
• convex Gaussian min-max theorem
• spectral estimator