Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data

2026-07-13

AI Summary

This work addresses the limitations of existing model compression methods, which are constrained by parameter count or data entropy and thus fail to accurately capture the actual information learned by a model. The authors propose “requential coding,” a novel approach based on a teacher–student architecture that self-generates training samples and applies information-theoretic encoding exclusively to instances where teacher and student predictions disagree. This mechanism achieves, for the first time, a compression scheme decoupled from both model size and data entropy. It substantially reduces code length—by several orders of magnitude compared to prequential coding—and delivers the strongest known PAC-Bayes generalization guarantees for billion-parameter language models. Furthermore, it accurately predicts overfitting trends across multiple training epochs and reveals that low-entropy text contains richer learnable structure than high-entropy images.

Abstract

Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.

Problem

• model compression
• prequential coding
• generalization
• learnable information
• code length

Innovation

• requential coding
• model compression
• PAC-Bayes bound
• self-generated training data
• generalization guarantee