Structure-aware divergences for comparing probability distributions

2026-03-23

AI Summary

This work proposes a class of structure-aware divergences that explicitly incorporate geometric relationships among elements in the support set of probability distributions—addressing a key limitation of classical information-theoretic measures such as Shannon entropy and f-divergences, which disregard structural similarities. By integrating the underlying geometry of the support set, the authors define a structure-aware entropy and derive corresponding Bregman divergences that retain desirable properties of the Kullback–Leibler divergence and Shannon entropy while embedding pairwise similarities directly into the divergence formulation. The approach successfully uncovers structural patterns missed by conventional methods in synthetic clustering tasks, achieves computational efficiency several orders of magnitude higher than optimal transport, and reproduces and extends established findings in applications to economic geography and ecology.

Technology Category

• Machine Learning: Information Theory
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Game Theory and Economic Paradigms: Fair Division

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Web Mining and Content Analysis: Bridging structured and unstructured data

Abstract

Many natural and social science systems are described using probability distributions over elements that are related to each other: for instance, occupations with shared skills or species with similar traits. Standard information theory quantities such as entropies and $f$-divergences treat elements interchangeably and are blind to the similarity structure. We introduce a family of divergences that are sensitive to the geometry of the underlying domain. By virtue of being the Bregman divergences of structure-aware entropies, they provide a framework that retains several advantages of Kullback-Leibler divergence and Shannon entropy. Structure-aware divergences recover planted patterns in a synthetic clustering task that conventional divergences miss and are orders of magnitude faster than optimal transport distances. We demonstrate their applicability in economic geography and ecology, where structure plays an important role. Modelling different notions of occupation relatedness yields qualitatively different regionalisations of their geographic distribution. Our methods also reproduce established insights into functional $β$-diversity in ecology obtained with optimal transport methods.

Problem

• structure-aware divergences
• probability distributions
• similarity structure
• f-divergences
• Shannon entropy

Innovation

• structure-aware divergences
• Bregman divergences
• probability distributions with geometry
• optimal transport
• functional β-diversity