Breaking the Finite-Sample Barrier in Entropy Coupling

2026-05-15

AI Summary

This work addresses the limitations of traditional entropy coupling models, which assume independent observations and struggle to eliminate residual uncertainty under finite samples. The authors propose a minimal list entropy coupling framework that permits arbitrary dependencies among observations while keeping marginal distributions fixed, thereby overcoming finite-sample constraints. By leveraging conditional entropy minimization, joint distribution construction, and support set analysis, they establish necessary and sufficient conditions—as well as structural criteria—for achieving zero-entropy couplings. A greedy algorithm with monotonic approximation guarantees is developed, which, under mild assumptions, attains zero residual uncertainty using only $O(\log(1/P_{\min}))$ samples. The framework is further extended to representation learning and randomness extraction tasks.

Abstract

Dependence among marginally constrained observations can break a finite-sample barrier. To formalize this phenomenon, we introduce the \emph{minimum list entropy coupling} $H(P\|Q_1,\dots,Q_m)$, the minimum conditional entropy $H(X|Y_1,\dots,Y_m)$ over all joint distributions with prescribed discrete marginals $X\sim P$ and $Y_i\sim Q_i$. Unlike classical formulations based on independent observations, our model allows $Y_1,\dots,Y_m$ to be arbitrarily dependent while keeping each marginal fixed. This enlarged coupling space reveals a sharp dichotomy: independent observations reduce residual uncertainty exponentially, whereas dependent observations can eliminate it exactly after finitely many samples. We characterize this zero-entropy regime through necessary and sufficient conditions and give concrete structural criteria under which it occurs. In particular, under mild support assumptions, zero entropy is achieved with $O(\log(1/P_{\min}))$ observations, where $P_{\min}$ is the minimum nonzero mass of $P$. We also develop a greedy algorithm with monotone approximation guarantees for computing $H(P\|Q_1,\dots,Q_m)$. Finally, we show that the same framework formalizes finite-sample limits in distribution-matching representation learning and randomness extraction, where zero entropy corresponds to exact recovery and exact extraction.

Problem

• entropy coupling
• finite-sample barrier
• marginal constraints
• conditional entropy
• zero-entropy regime

Innovation

• entropy coupling
• finite-sample barrier
• dependent observations
• zero-entropy regime
• distribution-matching representation learning