Generalized Score Matching: Bridging $f$-Divergence and Statistical Estimation Under Correlated Noise

2025-04-27

AI Summary

This paper investigates the quantitative relationship between estimation loss due to decoder mismatch and $f$-divergence for vector channels corrupted by anisotropic Gaussian noise, overcoming the limitations of prior work restricted to isotropic Gaussian assumptions and the relative entropy framework. Methodologically, we first extend score matching to anisotropic Gaussian channels, establishing a unified theory linking statistical estimation error to distributional discrepancy; we further generalize the De Bruijn identity to the mismatched setting and introduce the relative Fisher information to characterize the differential structure of $f$-divergence. Our main contribution is the derivation of an exact analytical expression connecting $f$-divergence to mismatched estimation error—providing a novel theoretical tool for generalization analysis and robust design of generative models.

Technology Category

• Machine Learning: Information Theory
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

Relative Fisher information, also known as score matching, is a recently introduced learning method for parameter estimation. Fundamental relations between relative entropy and score matching have been established in the literature for scalar and isotropic Gaussian channels. This paper demonstrates that such relations hold for a much larger class of observation models. We introduce the vector channel where the perturbation is non-isotropic Gaussian noise. For such channels, we derive new representations that connect the $f$-divergence between two distributions to the estimation loss induced by mismatch at the decoder. This approach not only unifies but also greatly extends existing results from both the isotropic Gaussian and classical relative entropy frameworks. Building on this generalization, we extend De Bruijn's identity to mismatched non-isotropic Gaussian models and demonstrate that the connections to generative models naturally follow as a consequence application of this new result.

Problem

• Extends score matching to non-isotropic Gaussian noise models
• Connects f-divergence to estimation loss in mismatched decoders
• Generalizes De Bruijn's identity for mismatched non-isotropic cases

Innovation

• Extends score matching to non-isotropic Gaussian noise
• Connects f-divergence to decoder mismatch loss
• Generalizes De Bruijn's identity for mismatched models