Singular Relative Entropy Coding with Bits-Back Rejection Sampling

2026-04-07

AI Summary

This work addresses the long-standing challenge in relative entropy coding of bridging the logarithmic redundancy gap between theoretical lower bounds and practical code lengths on singular channels. The authors propose Bitwise Backtracking Rejection Sampling (BBRS), a novel method that integrates bitwise backtracking coding with greedy rejection sampling. BBRS constitutes the first efficiently implementable coding scheme tailored to singular channels, achieving optimal sub-logarithmic asymptotic redundancy while significantly simplifying algorithmic structure, improving constant factors, and enhancing empirical performance.

Abstract

A relative entropy code for a source $X \sim P_X$ is a stochastic code that encodes random samples from a prescribed $P_{Y \mid X}$ using as few bits as possible. A generalisation of entropy coding, it is a standard result that the minimum number of bits required to achieve this is at least the mutual information $I[X\,\Vert\,Y]$. However, a particularly fascinating feature of relative entropy coding compared to entropy coding is that, in general, this lower bound is only achievable to within an additional logarithmic factor. As such, an important research direction is to identify channels where we can reduce this gap. Sriramu and Wagner achieved such success by exhibiting a relative entropy code for so-called singular channels with sub-logarithmic asymptotic redundancy. However, their code is quite involved and, sadly, cannot be implemented in practice. In this paper, we construct the bits-back rejection sampler (BBRS), a relative entropy code that combines ideas from bits-back coding and (greedy) rejection sampling. Our analysis of BBRS reveals that the algorithm achieves the same asymptotic efficiency as Sriramu and Wagner's sampler, but with much simpler analysis and better constants. Moreover, BBRS can be implemented using standard relative entropy coding methods.

Problem

• relative entropy coding
• singular channels
• mutual information
• asymptotic redundancy
• bits-back coding

Innovation

• relative entropy coding
• bits-back coding
• rejection sampling
• singular channels
• asymptotic redundancy