KL Divergence Between Gaussians: A Step-by-Step Derivation for the Variational Autoencoder Objective

2026-04-13

AI Summary

This study addresses the closed-form expression of the Kullback–Leibler (KL) divergence between Gaussian priors and posteriors in variational autoencoders (VAEs) and its role in training dynamics. Drawing on information theory and probability, the work systematically derives analytical solutions for the KL divergence under both univariate and diagonal-covariance multivariate Gaussian assumptions. It further elucidates how individual terms in the KL divergence contribute to regularization in the latent space and shape the model’s generative capacity. By providing a clear and rigorous theoretical derivation, this research deepens the understanding of the intrinsic nature of the VAE’s regularization term and offers principled guidance for model design and practical optimization.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Kullback-Leibler (KL) divergence is a fundamental concept in information theory that quantifies the discrepancy between two probability distributions. In the context of Variational Autoencoders (VAEs), it serves as a central regularization term, imposing structure on the latent space and thereby enabling the model to exhibit generative capabilities. In this work, we present a detailed derivation of the closed-form expression for the KL divergence between Gaussian distributions, a case of particular importance in practical VAE implementations. Starting from the general definition for continuous random variables, we derive the expression for the univariate case and extend it to the multivariate setting under the assumption of diagonal covariance. Finally, we discuss the interpretation of each term in the resulting expression and its impact on the training dynamics of the model.

Problem

• KL divergence
• Gaussian distributions
• Variational Autoencoder
• closed-form expression
• latent space regularization

Innovation

• KL divergence
• Variational Autoencoder
• Gaussian distribution
• closed-form derivation
• diagonal covariance