The Magnitude of Categories of Texts Enriched by Language Models

2025-01-11

AI Summary

This paper addresses the long-standing challenge of quantifying the “richness” of natural language text. We construct a [0,1]-enriched category structure grounded in token-level predictive probabilities from language models—marking the first integration of categorical enrichment and Magnitude theory into textual modeling. By embedding these probabilities into a generalized metric space, we define and derive an explicit closed-form expression for the text Magnitude function ( f(t) ). We prove that ( f'(1) ) exactly equals the sum of Shannon entropies across tokens and uncover its unification with Tsallis entropy; further, ( f(t) ) admits a physical interpretation as a statistical mechanical partition function. We fully characterize both zeroth- and first-order Magnitude homology groups. Collectively, this work establishes a novel theoretical bridge among linguistic information geometry, information theory, and categorical quantification, yielding a computationally tractable and interpretable mathematical framework for analyzing textual complexity.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Information Theory
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The purpose of this article is twofold. Firstly, we use the next-token probabilities given by a language model to explicitly define a $[0,1]$-enrichment of a category of texts in natural language, in the sense of Bradley, Terilla, and Vlassopoulos. We consider explicitly the terminating conditions for text generation and determine when the enrichment itself can be interpreted as a probability over texts. Secondly, we compute the M""obius function and the magnitude of an associated generalized metric space $mathcal{M}$ of texts using a combinatorial version of these quantities recently introduced by Vigneaux. The magnitude function $f(t)$ of $mathcal{M}$ is a sum over texts $x$ (prompts) of the Tsallis $t$-entropies of the next-token probability distributions $p(-|x)$ plus the cardinality of the model's possible outputs. The derivative of $f$ at $t=1$ recovers a sum of Shannon entropies, which justifies seeing magnitude as a partition function. Following Leinster and Schulman, we also express the magnitude function of $mathcal M$ as an Euler characteristic of magnitude homology and provide an explicit description of the zeroeth and first magnitude homology groups.

Problem

• Natural Language Richness Quantification
• Mathematical Space Properties
• Text Analysis

Innovation

• Language Model Probability
• Moebius Function
• Vigna's Method