NCO: A Versatile Plug-in for Handling Negative Constraints in Decoding

2026-05-11

AI Summary

This work addresses the challenge of efficiently avoiding multiple hard constraints—such as sensitive terms and personally identifiable information (PII)—as well as regular expression patterns during large language model (LLM) generation. Conventional automaton-based approaches often suffer from state-space explosion and high computational overhead. To overcome these limitations, the authors propose NCO decoding, a lightweight online pattern-matching strategy that dynamically enforces finite hard and regular-expression constraints without constructing large automata. NCO is the first method to enable online decoding control under multiple hard and regex constraints, seamlessly integrating with beam search and various sampling strategies. It further incorporates a soft masking mechanism to probabilistically suppress prohibited content. Experiments demonstrate that NCO significantly reduces violation rates in PII and profanity suppression tasks while preserving generation quality and inference efficiency.

Abstract

Controlling Large Language Models (LLMs) to prevent the generation of undesirable content, such as profanity and personally identifiable information (PII), has become increasingly critical. While earlier approaches relied on post-processing or resampling, recent research has shifted towards constrained decoding methods that control outputs during generation to mitigate high computational costs and quality degradation. However, preventing multiple forbidden hard constraints or regex constraints from appearing anywhere in the output is computationally challenging. A straightforward solution is to convert these constraints into a single automaton that tracks all forbidden patterns during decoding, but this often becomes impractically large. Standard regex engines also do not readily support the operations needed to build such a constraint, such as complement and intersection. In order to address these limitations, we propose NCO, a decoding strategy that performs online pattern matching over finite hard constraints and regex constraints, reducing computational overhead without inducing state explosion. NCO is fully compatible with standard inference strategies, including various sampling methods and beam search, while also supporting soft masking for probabilistic suppression. We empirically demonstrate its effectiveness across practical tasks, including PII and profanity suppression. Our implementation is available at https://github.com/hyundong98/NCO-Decoding.git .

Problem

• negative constraints
• constrained decoding
• large language models
• regex constraints
• undesirable content

Innovation

• constrained decoding
• negative constraints
• online pattern matching
• large language models
• regex constraints