Accelerating Constrained Decoding with Token Space Compression

2026-05-28

AI Summary

This work addresses the high latency of large language models in context-free grammar (CFG)-constrained decoding, which stems from the need to traverse the full vocabulary at each generation step, rendering complex grammars impractical. To overcome this limitation, the authors propose CFGzip, a novel offline method that compresses the token search space by integrating CFG analysis with semantic token clustering to construct a compact yet complete subset of valid tokens. This approach preserves generation correctness while drastically reducing the search scope. CFGzip seamlessly integrates with existing grammar-constrained decoding engines and, in standard evaluations, achieves up to a 7.5× speedup and reduces latency by two orders of magnitude, substantially enhancing the practicality and scalability of generating text under complex CFG constraints.

Abstract

To guarantee that an LLM's outputs conform to a specified structure, context-free grammar (CFG) decoding engines force the selection of next tokens that produce strings that conform to a given CFG. While current CFG-constrained decoding engines are highly optimized, the inherent costs arising from the massive per-step search space -- i.e. the entire token vocabulary -- result in intractably high overhead for more complex CFGs: precisely the situation where CFG engines are most useful. In this paper, we introduce CFGzip, an offline technique for compressing the token search space, which massively reduces CFG engine overhead. In experiments, we report latency reduction of up to two orders of magnitude when CFGzip is used with a SoTA grammar engine, yielding an up to 7.5x speedup in total constrained generation time: with CFGzip, constrained decoding is now feasible at scale for complex CFGs.

Problem

• constrained decoding
• context-free grammar
• token search space
• large language models
• decoding overhead

Innovation

• constrained decoding
• token space compression
• context-free grammar
• LLM optimization
• CFGzip