Loss-Complexity Landscape and Model Structure Functions

2025-07-17

AI Summary

This paper addresses the computational intractability of the Kolmogorov structure function (h_x(alpha)) by constructing a computationally tractable dual-theoretic framework. Methodologically, it establishes, for the first time, a Legendre–Fenchel duality between (h_x(alpha)) and a statistical-mechanical free-energy functional, modeling model complexity as an “energy” term parameterized by computable proxies (e.g., parameter norms, tree depth). Leveraging partition functions, Metropolis sampling, and detailed balance, it links information scattering magnitude to Markovian dynamics. Key contributions are: (1) identifying that the model complexity–generalization error phase transition is precisely characterized by a peak in “complexity susceptibility”; (2) proving this peak coincides exactly with the overfitting threshold and synchronizes strictly with the abrupt change in generalization error; and (3) empirically validating—on linear and tree regression models—a sharp, universal peak in complexity variance at the phase transition point.

Technology Category

• Machine Learning: Information Theory
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

We develop a framework for dualizing the Kolmogorov structure function $h_x(α)$, which then allows using computable complexity proxies. We establish a mathematical analogy between information-theoretic constructs and statistical mechanics, introducing a suitable partition function and free energy functional. We explicitly prove the Legendre-Fenchel duality between the structure function and free energy, showing detailed balance of the Metropolis kernel, and interpret acceptance probabilities as information-theoretic scattering amplitudes. A susceptibility-like variance of model complexity is shown to peak precisely at loss-complexity trade-offs interpreted as phase transitions. Practical experiments with linear and tree-based regression models verify these theoretical predictions, explicitly demonstrating the interplay between the model complexity, generalization, and overfitting threshold.

Problem

• Dualizing Kolmogorov structure function using computable complexity proxies
• Establishing analogy between information theory and statistical mechanics
• Investigating model complexity, generalization, and overfitting threshold

Innovation

• Dualizing Kolmogorov structure function with computable proxies
• Legendre-Fenchel duality between structure function and free energy
• Model complexity variance peaks at phase transitions