Draft-Conditioned Constrained Decoding for Structured Generation in LLMs

2026-02-08

arXiv.org

AI Summary

This work addresses the semantic drift often induced by hard constraints in structured generation, where conventional constrained decoding—while ensuring syntactic validity—can yield outputs that are locally well-formed yet semantically incorrect. To mitigate this issue, the authors propose a training-free, two-stage inference framework: first generating an unconstrained semantic draft, then performing constraint-aware decoding conditioned on this draft to decouple semantics from structure. By conditioning on the draft, the method alleviates the distributional shift caused by rigid constraints and optionally incorporates a multi-draft selection mechanism to further enhance output quality. Evaluated on benchmarks such as GSM8K, the approach improves structured accuracy by up to 24 percentage points (from 15.2% to 39.0%), demonstrating that ensembles of smaller models using this strategy can outperform larger models relying on traditional constrained decoding baselines.

Abstract

Large language models (LLMs) are increasingly used to generate executable outputs, JSON objects, and API calls, where a single syntax error can make the output unusable. Constrained decoding enforces validity token-by-token via masking and renormalization, but it can distort generation when the model assigns low probability mass to valid continuations, pushing decoding toward locally valid yet semantically incorrect trajectories. We propose \emph{Draft-Conditioned Constrained Decoding (DCCD)}, a simple two-step, training-free inference procedure that decouples semantic planning from structural enforcement: an unconstrained draft is generated first, and constrained decoding is then applied, conditioned on this draft, to guarantee validity. We analyze DCCD through a KL-projection view, showing that draft conditioning increases feasible mass and reduces the cumulative"projection tax"induced by hard constraints, with an optional best-of-$K$ draft selection. Across structured reasoning benchmarks, DCCD improves strict structured accuracy by up to +24 percentage points over standard constrained decoding (e.g., 15.2\% to 39.0\% on GSM8K with a 1B model), and enables smaller model pairs to match or exceed much larger constrained baselines, yielding substantial gains in parameter efficiency.

Problem

• structured generation
• constrained decoding
• large language models
• syntax errors
• semantic correctness

Innovation

• Constrained Decoding
• Draft-Conditioned Generation
• Structured Output
• Projection Tax
• Parameter Efficiency