Supervised Learning as Lossy Compression: Characterizing Generalization and Sample Complexity via Finite Blocklength Analysis

2026-02-04

AI Summary

This work investigates the generalization ability and sample complexity of supervised learning from an information-theoretic perspective, modeling the learning process as lossy compression with finite blocklength: training data sampling corresponds to encoding, and model construction to decoding. It introduces, for the first time, finite-blocklength lossy compression theory to analyze generalization error, deriving fundamental lower bounds on both generalization error and sample complexity for any fixed randomized learning algorithm and its optimal sampling strategy. The proposed framework cleanly disentangles the distinct contributions of overfitting and task-inductive bias mismatch, while unifying information-theoretic generalization bounds with the algorithmic stability perspective, thereby revealing their essential roles in determining generalization performance.

Technology Category

• Machine Learning: Information Theory
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

This paper presents a novel information-theoretic perspective on generalization in machine learning by framing the learning problem within the context of lossy compression and applying finite blocklength analysis. In our approach, the sampling of training data formally corresponds to an encoding process, and the model construction to a decoding process. By leveraging finite blocklength analysis, we derive lower bounds on sample complexity and generalization error for a fixed randomized learning algorithm and its associated optimal sampling strategy. Our bounds explicitly characterize the degree of overfitting of the learning algorithm and the mismatch between its inductive bias and the task as distinct terms. This separation provides a significant advantage over existing frameworks. Additionally, we decompose the overfitting term to show its theoretical connection to existing metrics found in information-theoretic bounds and stability theory, unifying these perspectives under our proposed framework.

Problem

• generalization
• sample complexity
• lossy compression
• overfitting
• inductive bias

Innovation

• lossy compression
• finite blocklength analysis
• generalization error
• sample complexity
• information-theoretic bounds