Reformulation Invariance and the Axiomatic Foundations of Inference

2026-06-19

AI Summary

This work addresses why the Kullback–Leibler (KL) divergence is uniquely suited for inference by formalizing inference as the selection of a minimal element within a preorder of positive measures, where divergences serve merely as numerical representations. Building on the axiom of reconstruction invariance—which requires that inference outcomes remain unchanged under equivalent problem formulations—the authors show that KL divergence emerges uniquely without invoking additional assumptions. This framework unifies maximum entropy, Bayesian updating, and exponential family estimation, extending classical axiomatic characterizations from finite alphabets to general measurable spaces. By integrating category theory, f-divergence theory, preorder structures, and Čencov’s category of statistical models, the paper establishes a rigorous mathematical foundation wherein inference operators arise naturally as covariant functors, applicable uniformly across both discrete and continuous settings.

Abstract

Maximum entropy, Bayesian updating, and exponential-family estimation are all instances of a common inference principle: selecting the measure or distribution that minimizes a divergence subject to the available constraints. Which divergence to use is usually decided by analytic convenience, by empirical performance, or by a set of axioms chosen to single it out, leaving open a basic question: why one divergence and not another? We answer it from a single requirement: an inference method should return the same answer whenever the same problem is presented in an equivalent form, for instance, after simply renaming its parts. This requirement alone forces inference to be the minimisation of a classical divergence, and each further reformulation it must respect tightens the admissible family one notch, narrowing the broad f-divergences to the α-divergences and finally to the single Kullback-Leibler (KL) divergence. Mathematically, inference is recast from minimising a numerical functional to selecting a least element under a preorder on positive measures, a divergence being merely one numerical scale that reproduces that preorder. The reformulations are the morphisms of a category of inference problems, and the invariance requirement says the inference operator is a covariant functor into the category of statistical models of Cencov, mirroring his characterisation of the Fisher metric. The representation is proved on finite spaces and lifted to general measurable spaces by an elementary closure, covering discrete and continuous spaces alike. Earlier axiomatisations, such as those of Shore-Johnson and Csiszar, postulate their consistency axioms directly and only on finite alphabets; here the axioms follow from reformulation invariance alone.

Problem

• divergence
• inference
• reformulation invariance
• Kullback-Leibler divergence
• axiomatic foundations

Innovation

• reformulation invariance
• Kullback-Leibler divergence
• f-divergence
• category theory
• statistical inference