Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression

2026-06-28

AI Summary

This work addresses the challenge of optimizing rate-distortion trade-offs in video compression below 0.1 bits per pixel (bpp), where existing methods struggle due to the absence of differentiable rate signals. The authors propose MS-VQ-VAE, a framework leveraging discrete latent variables and vector quantization combined with an autoregressive prior to model codebook usage distributions, enabling ultra-low-bitrate compression without explicit rate penalties. By revealing that codebook utilization follows a power-law distribution, they employ entropy coding to push empirical bitrates below theoretical limits. To stabilize training with small codebooks, they introduce exponential moving average (EMA) codebook updates and a dead-code revival mechanism, mitigating gradient collapse. On UCF101, the method achieves 0.043–0.064 bpp—5× to 7.6× more efficient than H.265—and consistently surpasses H.265 (CRF=36) in perceptual quality measured by LPIPS, with gains up to 0.072.

Abstract

Learned video codecs based on continuous latent representations struggle to operate reliably below 0.1 bits per pixel~(bpp): without a differentiable rate signal, Lagrangian optimisation cannot effectively trade reconstruction quality for bitrate at extreme compression ratios. We demonstrate that discrete latent representations sidestep this limitation entirely. In a vector-quantized~(VQ) codec, the codebook size~$K$ imposes a hard information ceiling of $\log_2 K$ bits per symbol; a learned autoregressive prior then exploits the non-uniform distribution of code usage -- which we show follows a power law -- to push actual bitrates well below this ceiling, without any rate-penalty tuning. Building on the MS-VQ-VAE architecture introduced in~\cite{kotthapalli2026msvqvae}, we sweep $K \in \{128, 256, 512, 1024\}$ under a uniform training protocol to trace four operating points on the rate-distortion~(RD) curve. We identify and resolve a critical training instability: gradient-based VQ collapses catastrophically at $K \leq 512$, whereas EMA-stabilised codebook updates with dead-code restart maintain full utilisation across all configurations. On 500 UCF101 test clips ($64\!\times\!64$, 32~frames), our models operate at 0.043-0.064~bpp -- 3.3-5$\times$ below H.264's practical floor and $5$-$7.6\times$ below H.265's floor at this resolution. Every MS-VQ-VAE configuration outperforms H.265 CRF\,36 on perceptual quality (LPIPS) despite using $5$-$7.6\times$ fewer bits. At $K{=}1024$, the model surpasses H.265 CRF\,36 on LPIPS by a margin of 0.072 absolute while using $5.1\times$ fewer bits. Codebook analysis confirms power-law index distributions and 70-85\% entropy efficiency, establishing the pipeline as a principled learned entropy coder.

Problem

• ultra-low bitrate
• video compression
• rate-distortion trade-off
• learned codecs
• discrete latent representations

Innovation

• discrete latent representation
• vector-quantized VAE
• learned autoregressive prior
• ultra-low bitrate compression
• entropy coding