Pool-based Active Learning as Noisy Lossy Compression: Characterizing Label Complexity via Finite Blocklength Analysis

2026-02-05

AI Summary

This work investigates the theoretical limits of label complexity and generalization error in pool-based active learning from an information-theoretic perspective. We model the problem as a noisy lossy compression process at finite blocklength, where data selection corresponds to encoding and the learning procedure to decoding, thereby introducing finite-blocklength information-theoretic tools into active learning analysis for the first time. By integrating stability theory with the lossy compression framework, we derive information-theoretic lower bounds on both label complexity and generalization error. These bounds elucidate the impact of inductive bias and overfitting on labeling efficiency and provide a fundamental characterization of the theoretical limits governing optimal data selection strategies.

Technology Category

• Machine Learning: Active Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper proposes an information-theoretic framework for analyzing the theoretical limits of pool-based active learning (AL), in which a subset of instances is selectively labeled. The proposed framework reformulates pool-based AL as a noisy lossy compression problem by mapping pool observations to noisy symbol observations, data selection to compression, and learning to decoding. This correspondence enables a unified information-theoretic analysis of data selection and learning in pool-based AL. Applying finite blocklength analysis of noisy lossy compression, we derive information-theoretic lower bounds on label complexity and generalization error that serve as theoretical limits for a given learning algorithm under its associated optimal data selection strategy. Specifically, our bounds include terms that reflect overfitting induced by the learning algorithm and the discrepancy between its inductive bias and the target task, and are closely related to established information-theoretic bounds and stability theory, which have not been previously applied to the analysis of pool-based AL. These properties yield a new theoretical perspective on pool-based AL.

Problem

• pool-based active learning
• label complexity
• noisy lossy compression
• finite blocklength analysis
• generalization error

Innovation

• active learning
• noisy lossy compression
• finite blocklength analysis
• label complexity
• information-theoretic bounds