Haiku to Opus in Just 10 bits: LLMs Unlock Massive Compression Gains

2026-02-09

AI Summary

This study addresses the challenge of efficiently compressing text generated by large language models (LLMs), achieving substantial improvements in compression ratios under both lossless and lossy settings. The authors propose a novel question-answering–based interactive compression protocol inspired by the “twenty questions” game, which narrows the performance gap between small and large models using only ten binary queries. By integrating domain-adapted LoRA fine-tuning, prompt rewriting, and arithmetic coding, the framework enables end-to-end optimization. Experimental results demonstrate a two-fold improvement in lossless compression efficiency, a lossy compression ratio of 0.03, and an exceptionally low interactive compression ratio ranging from 0.0006 to 0.004—surpassing existing methods by over two orders of magnitude.

Abstract

We study the compression of LLM-generated text across lossless and lossy regimes, characterizing a compression-compute frontier where more compression is possible at the cost of more compute. For lossless compression, domain-adapted LoRA adapters can improve LLM-based arithmetic coding by 2x over compression with the base LLM alone. For lossy compression, prompting a model for a succinct rewrite then applying arithmetic coding can achieve compression ratios of approximately 0.03, a 2x improvement over compressing the original response. We further introduce Question-Asking compression (QA), an interactive lossy protocol inspired by the game'Twenty Questions'. A small model iteratively refines its response by asking yes/no questions to a stronger model, transferring exactly one bit per answer. On 8 benchmarks spanning math, science, and code, 10 binary questions recover 23% to 72% of the capability gap between a small and large model on standard benchmarks and 7% to 38% on harder benchmarks, achieving compression ratios of 0.0006 to 0.004. This is over 100x smaller than prior LLM-based compression (Deletang et al., 2024), suggesting that interactive protocols can transfer knowledge far more efficiently than transmitting full responses.

Problem

• LLM compression
• lossy compression
• lossless compression
• compression ratio
• knowledge transfer

Innovation

• lossy compression
• arithmetic coding
• LoRA adaptation
• interactive protocol
• question-asking compression