DLM-SWAI: Steering Diffusion Language Models Before They Unmask

2026-05-28

AI Summary

Existing diffusion language models (DLMs) lack efficient, training-free methods for inference-time control that can effectively interface with their iterative denoising mechanism. This work proposes a training-free, token-level biasing approach that dynamically steers the generation distribution at each denoising step using precomputed style scores, thereby introducing token-level attribute cues into DLM inference for the first time. The method effectively guides text generation in style and safety control tasks while preserving high output quality, incurs minimal computational overhead, and enables a controllable trade-off between guidance strength and linguistic fluency. Furthermore, it reveals an intrinsic relationship between attribute steerability and the intensity of token-level cues.

Abstract

Steering language model generation toward desired textual properties is essential for practical deployment, and inference-time methods are particularly appealing because they enable controllable generation without retraining. Recent work has also highlighted diffusion language models as an emerging generation paradigm with distinct decoding properties. However, most existing steering approaches either rely on auxiliary models or are designed for autoregressive next-token decoding, making them difficult to apply to diffusion language models DLMs, which generate text through iterative denoising of partially masked sequences. Therefore, we propose DLM-SWAI, a simple training-free steering method that biases the token distribution at each denoising step using pre-computed token-level style scores. Experiments on style and safety control tasks show that DLM-SWAI effectively steers diffusion language models while preserving generation quality and requiring minimal computational overhead. Ablations further reveal a controllable trade-off between steering strength and fluency, and our analysis links class-wise steerability to the strength of token-level attribute cues.

Problem

• diffusion language models
• steering
• inference-time control
• text generation
• controllable generation

Innovation

• diffusion language models
• inference-time steering
• training-free control
• token-level style scores
• iterative denoising