Optimal Stability of KL Divergence under Gaussian Perturbations

2026-04-13

AI Summary

This work investigates the stability of Kullback–Leibler (KL) divergence under Gaussian perturbations, extending beyond prior studies restricted to Gaussian distributions. For any distribution \( P \) with finite second moments and Gaussian distributions \( N_1 \) and \( N_2 \), it establishes that if \( \mathrm{KL}(P \| N_1) \) is large and \( \mathrm{KL}(N_1 \| N_2) \leq \varepsilon \), then \( \mathrm{KL}(P \| N_2) \geq \mathrm{KL}(P \| N_1) - O(\sqrt{\varepsilon}) \). The key contribution lies in generalizing the relaxed triangle inequality for KL divergence from the Gaussian family to arbitrary distributions with finite variance, while proving the optimality of the \( \sqrt{\varepsilon} \) rate. This result provides rigorous theoretical support for out-of-distribution detection in flow-based models under non-Gaussian settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

We study the problem of characterizing the stability of Kullback-Leibler (KL) divergence under Gaussian perturbations beyond Gaussian families. Existing relaxed triangle inequalities for KL divergence critically rely on the assumption that all involved distributions are Gaussian, which limits their applicability in modern applications such as out-of-distribution (OOD) detection with flow-based generative models. In this paper, we remove this restriction by establishing a sharp stability bound between an arbitrary distribution and Gaussian families under mild moment conditions. Specifically, let $P$ be a distribution with finite second moment, and let $\mathcal{N}_1$ and $\mathcal{N}_2$ be multivariate Gaussian distributions. We show that if $KL(P||\mathcal{N}_1)$ is large and $KL(\mathcal{N}_1||\mathcal{N}_2)$ is at most $ε$, then $KL(P||\mathcal{N}_2) \ge KL(P||\mathcal{N}_1) - O(\sqrtε)$. Moreover, we prove that this $\sqrtε$ rate is optimal in general, even within the Gaussian family. This result reveals an intrinsic stability property of KL divergence under Gaussian perturbations, extending classical Gaussian-only relaxed triangle inequalities to general distributions. The result is non-trivial due to the asymmetry of KL divergence and the absence of a triangle inequality in general probability spaces. As an application, we provide a rigorous foundation for KL-based OOD analysis in flow-based models, removing strong Gaussian assumptions used in prior work. More broadly, our result enables KL-based reasoning in non-Gaussian settings arising in deep learning and reinforcement learning.

Problem

• KL divergence
• Gaussian perturbations
• stability
• out-of-distribution detection
• relaxed triangle inequality

Innovation

• KL divergence
• Gaussian perturbations
• stability bound
• relaxed triangle inequality
• out-of-distribution detection