Tighter CMI-Based Generalization Bounds via Stochastic Projection and Quantization

2025-10-27

AI Summary

This work addresses two fundamental questions in statistical learning: (i) the looseness of existing conditional mutual information (CMI)-based generalization bounds, and (ii) whether “data memorization” is necessary for good generalization. We propose a novel CMI analysis framework grounded in random projection and lossy quantization, overcoming the failure of conventional CMI bounds in high-dimensional or structured settings. Theoretically, we prove that memorizing most training samples is *not* necessary for generalization; furthermore, we construct an auxiliary algorithm achieving an optimal $O(1/sqrt{n})$ generalization bound without significant memorization. This constitutes the first information-theoretic characterization of the intrinsic relationship between generalization and memorization. Crucially, our framework preserves generalization performance while substantially reducing memory reliance on training data—providing both theoretical foundations and new analytical tools for designing efficient, low-memory learning algorithms.

Technology Category

• Machine Learning: Information Theory
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In this paper, we leverage stochastic projection and lossy compression to establish new conditional mutual information (CMI) bounds on the generalization error of statistical learning algorithms. It is shown that these bounds are generally tighter than the existing ones. In particular, we prove that for certain problem instances for which existing MI and CMI bounds were recently shown in Attias et al. [2024] and Livni [2023] to become vacuous or fail to describe the right generalization behavior, our bounds yield suitable generalization guarantees of the order of $mathcal{O}(1/sqrt{n})$, where $n$ is the size of the training dataset. Furthermore, we use our bounds to investigate the problem of data "memorization" raised in those works, and which asserts that there are learning problem instances for which any learning algorithm that has good prediction there exist distributions under which the algorithm must "memorize" a big fraction of the training dataset. We show that for every learning algorithm, there exists an auxiliary algorithm that does not memorize and which yields comparable generalization error for any data distribution. In part, this shows that memorization is not necessary for good generalization.

Problem

• Establish tighter CMI bounds for generalization error
• Address data memorization issues in learning algorithms
• Prove memorization is unnecessary for good generalization

Innovation

• Stochastic projection and lossy compression techniques
• Tighter conditional mutual information generalization bounds
• Auxiliary algorithm achieving generalization without memorization