Stein's unbiased risk estimate and Hyv""arinen's score matching

2025-02-27

AI Summary

This paper investigates empirical Bayesian denoising of signals corrupted by Gaussian noise. Addressing the challenge of unknown signal distributions, it proposes two unsupervised prior estimation strategies: (i) optimizing the Tweedie denoiser via Stein’s Unbiased Risk Estimate (SURE), and (ii) minimizing Fisher divergence through Hyvärinen score matching. The work establishes, for the first time, a unified theoretical framework linking SURE and score matching under both well-specified and misspecified models, yielding near-parametric convergence rates and oracle inequalities. Theory and experiments demonstrate that, under correct model specification, the methods match the performance of nonparametric maximum likelihood; under misspecification—e.g., heteroscedastic noise or covariate-dependent structures—they exhibit superior robustness and faster convergence. The core contributions lie in this unifying analytical perspective and the ability to adaptively model complex noise structures without explicit parametric assumptions.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We study two G-modeling strategies for estimating the signal distribution (the empirical Bayesian's prior) from observations corrupted with normal noise. First, we choose the signal distribution by minimizing Stein's unbiased risk estimate (SURE) of the implied Eddington/Tweedie Bayes denoiser, an approach motivated by optimal empirical Bayesian shrinkage estimation of the signals. Second, we select the signal distribution by minimizing Hyv""arinen's score matching objective for the implied score (derivative of log-marginal density), targeting minimal Fisher divergence between estimated and true marginal densities. While these strategies appear distinct, they are known to be mathematically equivalent. We provide a unified analysis of SURE and score matching under both well-specified signal distribution classes and misspecification. In the classical well-specified setting with homoscedastic noise and compactly supported signal distribution, we establish nearly parametric rates of convergence of the empirical Bayes regret and the Fisher divergence. In a commonly studied misspecified model, we establish fast rates of convergence to the oracle denoiser and corresponding oracle inequalities. Our empirical results demonstrate competitiveness with nonparametric maximum likelihood in well-specified settings, while showing superior performance under misspecification, particularly in settings involving heteroscedasticity and side information.

Problem

• Learning optimal denoising methods for Gaussian noise corrupted signals
• Comparing SURE minimization with NPMLE in empirical Bayes settings
• Developing practical implementations for heteroscedastic noise with side-information

Innovation

• SURE minimization achieves parametric convergence rates
• SURE-training converges fast to oracle denoiser
• Practical implementation with heteroscedasticity and side-information