On the Query Complexity of Verifier-Assisted Language Generation

2025-02-17

AI Summary

This paper addresses the challenge of simultaneously achieving high generation quality and computational efficiency in constrained language generation. We propose a collaborative generation framework augmented with a *process verifier*—a novel component formally defined for the first time. We prove that such a verifier can transform information-theoretically or computationally hard constrained generation problems into ones solvable in polynomial time. To operationalize this, we design a token-level rejection sampling algorithm with backtracking support, overcoming the limitations of conventional block-level sampling. Theoretical analysis characterizes the verifier’s mechanism for compressing query complexity. Empirical evaluation demonstrates that our method significantly outperforms both block-wise rejection sampling and nucleus sampling on tasks including mathematical reasoning and program synthesis—achieving higher accuracy, improved computational efficiency, and greater output diversity.

Technology Category

• Natural Language Processing: Generation
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recently, a plethora of works have proposed inference-time algorithms (e.g. best-of-n), which incorporate verifiers to assist the generation process. Their quality-efficiency trade-offs have been empirically benchmarked on a variety of constrained generation tasks, but the algorithmic design landscape is still largely poorly understood. In this paper, we develop a mathematical framework for reasoning about constrained generation using a pre-trained language model generator oracle and a process verifier--which can decide whether a prefix can be extended to a string which satisfies the constraints of choice. We show that even in very simple settings, access to a verifier can render an intractable problem (information-theoretically or computationally) to a tractable one. In fact, we show even simple algorithms, like tokenwise rejection sampling, can enjoy significant benefits from access to a verifier. Empirically, we show that a natural modification of tokenwise rejection sampling, in which the sampler is allowed to"backtrack"(i.e., erase the final few generated tokens) has robust and substantive benefits over natural baselines (e.g. (blockwise) rejection sampling, nucleus sampling)--both in terms of computational efficiency, accuracy and diversity.

Problem

• Develops framework for constrained generation
• Verifiers transform intractable to tractable problems
• Backtracking enhances efficiency, accuracy, diversity

Innovation

• Verifier-assisted generation process
• Tokenwise rejection sampling
• Backtracking modification benefits