From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence

2026-01-06

arXiv.org

AI Summary

This work addresses a key limitation of classical information theory—such as Shannon entropy and Kolmogorov complexity—which assumes observers possess unbounded computational resources and thus fails to capture the useful information accessible to computationally constrained agents. To bridge this gap, the paper introduces **epiplexity**, the first structured information measure tailored for resource-bounded learners. Epiplexity formally quantifies the information content learnable within explicit time constraints, explicitly excluding random entropy that cannot be exploited. Grounded in computational complexity theory, information theory, and machine learning, this framework reveals that information can be computationally generated, is sensitive to data ordering, and offers a theoretical foundation for principled data selection. Empirical results demonstrate that epiplexity effectively discriminates between data sources, correlates strongly with downstream task performance, and informs data intervention strategies that enhance out-of-distribution generalization.

Technology Category

• Machine Learning: Information Theory
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Game Theory and Economic Paradigms: Imperfect Information

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Data transparency and provenance

Abstract

Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated without considering a downstream task? On these questions, Shannon information and Kolmogorov complexity come up nearly empty-handed, in part because they assume observers with unlimited computational capacity and fail to target the useful information content. In this work, we identify and exemplify three seeming paradoxes in information theory: (1) information cannot be increased by deterministic transformations; (2) information is independent of the order of data; (3) likelihood modeling is merely distribution matching. To shed light on the tension between these results and modern practice, and to quantify the value of data, we introduce epiplexity, a formalization of information capturing what computationally bounded observers can learn from data. Epiplexity captures the structural content in data while excluding time-bounded entropy, the random unpredictable content exemplified by pseudorandom number generators and chaotic dynamical systems. With these concepts, we demonstrate how information can be created with computation, how it depends on the ordering of the data, and how likelihood modeling can produce more complex programs than present in the data generating process itself. We also present practical procedures to estimate epiplexity which we show capture differences across data sources, track with downstream performance, and highlight dataset interventions that improve out-of-distribution generalization. In contrast to principles of model selection, epiplexity provides a theoretical foundation for data selection, guiding how to select, generate, or transform data for learning systems.

Problem

• information theory
• computationally bounded intelligence
• useful information
• data value
• epiplexity

Innovation

• epiplexity
• computationally bounded intelligence
• information theory
• data selection
• structural information