Real-Time Text Transmission via LLM-Based Entropy Coding over Fixed-Rate Channels

2026-05-03

AI Summary

This study addresses the trade-off between compression efficiency and transmission delay when using variable-length coding for real-time text streaming over fixed-rate channels, where queueing delays arise from fluctuating codeword lengths. For the first time, it systematically evaluates predictive coding driven by large language models in this context, employing GPT-2 (124M) and Llama 3.2 (3B) as causal predictors combined with entropy coding schemes including Shannon, Huffman, arithmetic coding, rANS, and gzip. Experimental results demonstrate that increasing model size by a factor of 25 reduces bits per character by approximately 38%. Among entropy coders, Huffman coding proves more practical in over-provisioned channels due to its zero algorithmic delay, whereas arithmetic coding approaches the theoretical compression limit at the cost of introducing decoding latency.

Abstract

Learning, prediction, and compression are intimately connected: a model that accurately predicts the next symbol in a sequence can be coupled with a source coder to compress that sequence near its information-theoretic limit. When tokenized characters arriving at a fixed reading pace are encoded into variable-length codewords and streamed over a fixed-rate channel, a queue forms whose per-token delay depends on the mean and variance of the bit lengths and on the coder's algorithmic latency. This paper investigates the compression--delay tradeoff that arises when a causal language model serves as the sequential predictor within a predict-then-code architecture for real-time text transmission. Several coding schemes are compared: Shannon (ideal), Huffman, arithmetic coding, rANS at various block sizes, and gzip. The analysis separates algorithmic delay, inherent to the coder, from computational delay, which shrinks as hardware improves. Huffman is the practical choice for over-provisioned channels, with zero algorithmic delay and modest compression overhead. Arithmetic coding achieves near-optimal compression at the cost of decodability delay. Findings are validated across two scales: GPT-2 (124M) and Llama~3.2 (3B), a twenty-five-fold parameter range. This scaling yields an approximately 38\% reduction in bits per character, effectively over-provisioning the channel and thereby changing which coder is optimal.

Problem

• real-time text transmission
• entropy coding
• compression-delay tradeoff
• fixed-rate channels
• causal language model

Innovation

• LLM-based entropy coding
• compression-delay tradeoff
• real-time text transmission
• predict-then-code architecture
• fixed-rate channels