Trie Automata for Constrained Decoding over Large Finite Sets

2026-08-12

AI Summary

This work addresses the performance bottleneck—termed the “cardinality wall”—faced by existing grammar-constrained decoding methods for large language models when generating structured outputs conforming to predefined schemas, particularly when the candidate set size is large (K ≥ 300). The authors propose a novel automaton-based mechanism that integrates prefix trees (Tries) with the Aho-Corasick multi-pattern matching algorithm. By leveraging shared prefixes, bounded depth, and known cardinality of the candidate set, the method precomputes valid token masks at each trie node, enabling stateless and highly efficient constrained decoding. This approach, the first to combine Trie and Aho-Corasick techniques for decoding constraints, achieves sub-100ms compilation time even at K = 10,000, offering 2–6.5× faster compilation and 7× lower per-token latency (0.65 μs vs. 5.8 μs) than XGrammar. At batch size 256, it attains an end-to-end throughput of 219 requests/second—a 29× improvement—while guaranteeing 100% output validity.

Abstract

Large language models increasingly need to generate structured outputs that conform to predefined schemas, with one common constraint being selection from a finite set of valid strings. Current constrained decoding systems handle this through general-purpose grammar compilation, which becomes prohibitively slow as the number of valid values grows into the thousands, a cardinality wall. We introduce the trie automaton, a specialized mechanism that exploits finite-set structure (shared prefixes, bounded depth, known cardinality) via Aho-Corasick multi-pattern matching to precompute per-node token masks. The trie achieves 7X faster per-step valid-token computation (0.65 us vs. 5.8 us) compared to XGrammar, one of the primary backends in vLLM and SGLang, and 2--6.5X faster compilation at K >= 300. Because precomputed masks enable a stateless serving path that bypasses the guided decoding pipeline, this advantage compounds in batch serving: end-to-end vLLM throughput reaches 219 req/s vs. XGrammar's 7.5 req/s at batch size 256 (29X). The 29X combines the algorithmic speedup with integration-path savings that only precomputed masks can unlock. Across seven tokenizer families (32K--262K vocabulary), the trie maintains sub-100ms compilation up to K = 10,000 and flat per-step cost regardless of set size, while guaranteeing 100% output validity.

Problem

• constrained decoding
• large finite sets
• structured output
• grammar compilation
• cardinality wall

Innovation

• trie automaton
• constrained decoding
• Aho-Corasick
• precomputed token masks
• large finite sets