The Jacobian and Hessian of the Kullback-Leibler Divergence between Multivariate Gaussian Distributions (Technical Report)

2025-06-30

AI Summary

This paper addresses the analytical computation of first- and second-order derivatives—specifically, the Jacobian and Hessian matrices—of the Kullback–Leibler (KL) divergence between multivariate Gaussian distributions with respect to their mean vectors and symmetric positive-definite covariance matrices. Leveraging matrix differential calculus, we unify the Magnus differential framework with Minka’s technique to derive, for the first time, closed-form expressions for all first- and second-order derivatives of the KL divergence with respect to both parameters. The derivation is mathematically rigorous, fully transparent, and pedagogically interpretable. The resulting formulae substantially enhance the computability and interpretability of gradient and curvature information in variational inference, information geometry, and probabilistic model optimization. By providing exact, compact second-order derivatives, this work furnishes a critical mathematical foundation for developing efficient, curvature-aware optimization algorithms in statistical machine learning.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This document shows how to obtain the Jacobian and Hessian matrices of the Kullback-Leibler divergence between two multivariate Gaussian distributions, using the first and second-order differentials. The presented derivations are based on the theory presented by cite{magnus99}. I've also got great inspiration from some of the derivations in cite{minka}. Since I pretend to be at most didactic, the document is split into a summary of results and detailed derivations on each of the elements involved, with specific references to the tricks used in the derivations, and to many of the underlying concepts.

Problem

• Compute Jacobian of KL divergence for Gaussians
• Derive Hessian of KL divergence for Gaussians
• Explain differential methods for matrix calculus

Innovation

• Jacobian and Hessian of KL divergence
• First and second-order differentials
• Multivariate Gaussian distributions