Rejection Sampling is Optimal for Relative Entropy Coding

2026-04-24

AI Summary

In relative entropy coding, conventional mutual information lower bounds suffer from an irreducible logarithmic gap. This work proposes “looped投 coding” based on rejection sampling and establishes, for the first time, that the functional information content constitutes a tight lower bound for this task, thereby proving the optimality of rejection sampling. The constructed scheme achieves a communication cost of at most the functional information plus $\log e$ bits in the one-shot setting. Furthermore, it reduces the redundancy to $\log(I+1) + 2.45$ bits for general channels and to $1.45$ bits for singular channels, reproducing and strengthening existing asymptotic second-order redundancy results.

Technology Category

• Machine Learning: Information Theory
• Reasoning under Uncertainty: Stochastic Optimization
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Web Mining and Content Analysis: Content-based information diffusion
• Security and Privacy: Applications of cryptography
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In relative entropy coding, a sender aims to design a stochastic code such that, on input $X \sim P_X$, the receiver can generate a sample $Y \sim P_{Y \mid X}$. It is a standard result that (1) this requires at least $I(X; Y)$ bits, (2) the lower bound is achievable within a logarithmic gap, and (3) this gap cannot be reduced in general. The necessity of the gap suggests that the mutual information is not the correct information measure to quantify the rate of relative entropy coding. A potential alternative emerged in the work of Flamich et al. (2025), who proved a tighter lower bound of $I_F(X \to Y)$, a quantity we call the functional information. In this paper, we show that this lower bound is tight by constructing the ring toss code, an encoding method for rejection sampling which uses at most $I_F(X \to Y) + \log e$ bits. This demonstrates that rejection sampling is optimal for relative entropy coding. Our result implies that the classical mutual information lower bound is achievable within $\log(I(X; Y) + 1) + 2.45$ bits in general and within $1.45$ bits for singular channels, which are both the tightest bounds of their kind to date. Moreover, our one-shot result also recovers Sriramu and Wagner's asymptotic results on the second-order redundancy of relative entropy codes.

Problem

• relative entropy coding
• mutual information
• functional information
• rejection sampling
• information-theoretic lower bound

Innovation

• rejection sampling
• relative entropy coding
• functional information
• ring toss code
• one-shot coding