On Unified and Sharpened CMI Bounds for Generalization Errors

2026-05-20

AI Summary

Existing generalization error bounds based on conditional mutual information (CMI) suffer from redundancy and looseness. This work proposes a unified CMI framework grounded in the leave-$m$-out hyper-sample cross-validation error, which achieves tighter bounds by reducing the number of conditioning terms. The framework not only encompasses and reproduces several known CMI-based bounds but also bridges the gap between classical mutual information (MI) and CMI bounds in the limit as $m \to \infty$. Theoretical analysis and multiple illustrative examples demonstrate that the proposed bound is superior to existing results in terms of tightness, applicability, and expressiveness.

Abstract

We present a new family of information-theoretic generalization bounds within the framework of conditional mutual information (CMI). Most of our results are established based on the leave-$m$-out (L$m$O) cross-validation error, with $m$ denoting the number of the hold-out supersamples. Under this setting, we propose a unified CMI-based bound, allowing to envelop and reproduce many known CMI-based bounds and also bridge the gap between the MI- and CMI-based bounds when $m$ tends to infinity. The proposed framework not only provides a unified description of the existing bounds but also develops new, sharper bounds. We show the benefits of the proposed bounds through several simple examples, where the existing results are either inapplicable or looser. Moreover, under the premise that the loss function is bounded, we tighten the CMI quantities involved in the proposed bounds by reducing the number of conditional terms, thereby enhancing the proposed framework. We show empirically that the resulting new bounds improve upon the previously known ones.

Problem

• generalization error
• conditional mutual information
• information-theoretic bounds
• cross-validation
• tighter bounds

Innovation

• conditional mutual information
• generalization bounds
• leave-m-out cross-validation
• information-theoretic learning
• sharper bounds