Complete Diagrammatic Axiomatisations of Relative Entropy

2026-03-04

arXiv.org

AI Summary

This work presents an axiomatic study of relative entropy—including Kullback–Leibler divergence and Rényi divergences of arbitrary order—from a categorical perspective. By endowing the category of stochastic matrices with a quantale-enriched structure, the authors develop a diagrammatic axiomatization grounded in quantale-valued monoidal algebras, tailored to the two natural monoidal structures given by the Kronecker product and direct sum. The paper innovatively provides, for the first time, a complete diagrammatic axiomatization of both families of relative entropy under these dual monoidal structures. Integrating tools from category theory, information geometry, and string diagram syntax, this approach establishes a unified and formally rigorous framework that lays the theoretical foundation for algebraic and diagrammatic reasoning about distances between probability distributions.

Abstract

Relative entropy is a fundamental class of distances between probability distributions, with widespread applications in probability theory, statistics, and machine learning. In this work, we study relative entropy from a categorical perspective, viewing it as a quantitative enrichment of categories of stochastic matrices. We consider two natural monoidal structures on stochastic matrices, given by the Kronecker product and the direct sum. Our main results are complete axiomatisations of Kullback-Leibler divergence and, more generally, of R\'enyi divergences of arbitrary order, for each such structure. Our axiomatic theories are formulated within the framework of quantitative monoidal algebra, using a graphical language of string diagrams enriched with quantitative equations.

Problem

• relative entropy
• Kullback-Leibler divergence
• Rényi divergence
• stochastic matrices
• monoidal structure

Innovation

• relative entropy
• categorical axiomatisation
• string diagrams
• quantitative monoidal algebra
• Rényi divergences