Characterizing Learning in Deep Neural Networks using Tractable Algorithmic Complexity Analysis

2026-05-14

AI Summary

Existing methods for estimating algorithmic complexity struggle to scale to large-scale non-binary deep neural networks and fail to effectively capture the interplay between learning and compression during training. This work proposes Quantized Block Decomposition (QuBD), a novel approach that integrates weight quantization, bit-plane decomposition, and complexity estimation based on the Coding Theorem Method (CTM) and Block Decomposition Method (BDM), enabling the first scalable Kolmogorov–Chaitin–Solomonoff (KCS) complexity analysis of large non-binary weights. The study reveals that algorithmic information is predominantly concentrated in higher-order bit planes; weight complexity consistently decreases during training and correlates strongly with dataset size and generalization performance; and this complexity measure can effectively guide post-training quantization, exhibiting consistent patterns in both overfitting and grokking phenomena.

Abstract

Training large-scale deep neural networks (DNNs) is resource-intensive, making model compression a practical necessity. The widely accepted ''learning as compression'' hypothesis posits that training induces structure in network weights, which enables compression. Measuring this structure through Kolmogorov-Chaitin-Solomonoff (KCS) complexity is appealing, but existing estimators based on the Coding Theorem Method (CTM) and the Block Decomposition Method (BDM) are limited to small binary objects and do not scale to modern DNNs. We introduce the Quantized Block Decomposition method (QuBD), which extends algorithmic complexity estimation to any $k$-ary object. QuBD first quantizes the network weights to a finite alphabet, then estimates the KCS complexity by aggregating per bit-plane CTM estimates. We show theoretically that QuBD yields a strictly tighter estimation gap with respect to true KCS complexity than binarization-based methods. Using QuBD, we study how the algorithmic complexity of neural network weights evolves during training, showing that it decreases as models learn, scales with data budget, increases during overfitting, follows the delayed generalization observed during grokking, and correlates with generalization performance. We further show that algorithmic information resides predominantly in the most significant bit-planes, which can serve as a practical diagnostic for determining appropriate post-training quantization levels. This work offers novel insights into learning mechanisms in DNNs by providing the first scalable, tractable estimates of KCS complexity for large, non-binary objects such as DNN weights.

Problem

• algorithmic complexity
• deep neural networks
• Kolmogorov complexity
• model compression
• weight structure

Innovation

• algorithmic complexity
• Quantized Block Decomposition
• Kolmogorov complexity
• model compression
• bit-plane analysis