Diffuse to Compress: Leveraging Diffusion LMs for Lossless Compression

2026-08-04

AI Summary

This work addresses the throughput bottleneck in lossless text compression based on neural language models by introducing, for the first time, diffusion language models (DLMs) into this framework. Replacing the conventional autoregressive symbol-by-symbol generation with a non-autoregressive approach, the proposed method substantially enhances compression efficiency. To tackle the algorithmic challenges posed by DLMs in lossless compression, the authors devise an effective dynamic strategy for selecting symbol positions and counts. Experimental results on the enwik8 benchmark demonstrate that the method not only surpasses existing large language models and general-purpose compressors such as zstd and gzip but also achieves a synergistic optimization of compression speed and ratio, establishing a new performance frontier while exhibiting strong scalability.

Abstract

We study the problem of lossless text compression, motivated by the rapid growth in the collection and storage of digital textual data - including plain text, source code, and structured formats such as XML - and by recent advances in neural language model-based compression. In particular, recent LLM-based approaches, whether built on symbol-ranking pipelines or paired with a statistical compressor, have demonstrated compression ratios significantly superior to general-purpose compressors such as zstd, gzip, or bzip on text and code. However, these neural approaches suffer from severe throughput limitations, making them not yet practically usable. For the first time in the context of lossless neural text compression, we introduce Diffusion Language Models (DLMs) as an alternative inference paradigm to autoregressive LLM-based approaches. We argue that replacing autoregressive LLMs with DLMs within the same compression framework could overcome the throughput bottleneck caused by their one-symbol-per-step limitation. However, achieving these improvements requires addressing algorithmic challenges introduced by applying DLMs to lossless compression, where the architecture allows the number and positions of symbols encoded at each forward pass to be decided independently. We design efficient and effective strategies to solve these challenges and evaluate them experimentally against LLM-based and general-purpose compressors on enwik8, a well-established textual benchmark. Our results show that the newly proposed DLM-based framework advances the state of the art in lossless text compression. Moreover, as DLMs are still a relatively young paradigm, recent advances toward increasingly capable and efficient models suggest substantial room for further improvements.

Problem

• lossless compression
• neural language models
• throughput bottleneck
• text compression
• Diffusion Language Models

Innovation

• Diffusion Language Models
• lossless compression
• throughput optimization
• non-autoregressive inference
• neural text compression