Learning the Error Patterns of Language Models

2026-05-27

AI Summary

Large language models often struggle to generate valid constrained outputs—such as compilable code—due to recurring error patterns like syntactic ambiguities. To address this, this work proposes the “prefix filter,” a domain- and model-specific symbolic function designed to characterize and correct such errors, along with Palla, an efficient learning algorithm that integrates symbolic constraint representations with constraint-guided sampling. By leveraging this approach, the method substantially improves generation quality. Evaluated on TypeScript code generation, Palla boosts the compilation success rate of Qwen2.5-1.5B by over 60%, achieving performance comparable to that of the much larger, unconstrained Llama3.1-8B model.

Abstract

When generating outputs for domains with specific validity constraints (e.g., a program should compile), LLMs often fail in a small number of focused ways: for example, by using Python function names when generating TypeScript. We observe that these error patterns can be represented using a small number of constraints that can be learned in practice. We propose \emph{prefix filters}, which are per-domain-and-LLM symbolic functions, as objects to capture the error patterns, Palla as an algorithm to learn prefix filters efficiently in practice, and implement Palla. Prefix filters learned by Palla i) help us quantitatively analyze the error patterns of LLMs, and ii) can be used to constrain the outputs of a model via constrained sampling algorithms. For example, Palla boosts compile rates for Qwen2.5-1.5B on TypeScript generation, by over 60%, allowing Qwen2.5-1.5B to achieve similar performance to Llama3.1-8B unconstrained.

Problem

• error patterns
• language models
• validity constraints
• code generation
• domain-specific errors

Innovation

• prefix filters
• error pattern learning
• constrained sampling
• Palla algorithm
• domain-specific validity