Guidance Is Not a Hyperparameter: Learning Dynamic Control in Diffusion Language Models

2026-05-08

AI Summary

This work addresses the limitations of existing Classifier-Free Guidance, which relies on a static guidance scale and struggles to balance the dynamic trade-offs between controllability and generation quality across different tasks and diffusion timesteps. To overcome this, the paper formulates guidance as a sequential decision-making problem and introduces a policy network trained via Proximal Policy Optimization (PPO) to enable task-adaptive, dynamic guidance control within discrete diffusion language models. By moving beyond fixed hyperparameters, the proposed approach significantly outperforms static guidance strategies across three controllable text generation benchmarks, effectively harmonizing output quality with task-specific control requirements.

Abstract

Classifier-Free Guidance (CFG) is a widely used mechanism for controlling diffusion-based generative models, yet its guidance scale is typically treated as a fixed hyperparameter throughout generation. This static design yields a suboptimal controllability and quality tradeoff, as the optimal degree of guidance varies across tasks and across different stages of the diffusion process, especially in NLP domain. We recast CFG scale selection as a sequential decision-making problem and propose to learn dynamic guidance trajectories via reinforcement learning. Specifically, we model the guidance scale as a discrete control action selected at each generation step based on the evolving diffusion state, and optimize a policy using Proximal Policy Optimization (PPO) under task-level rewards. Experiments on three controlled NLP generation tasks using discrete diffusion language models demonstrate that adaptive guidance consistently achieves a better balance between controllability and generation quality than fixed-scale strategies. Further analysis of the learned policies reveals distinct and interpretable guidance trajectories across tasks, underscoring the importance of treating guidance as a dynamic control process rather than a static design choice.

Problem

• Classifier-Free Guidance
• diffusion language models
• dynamic control
• guidance scale
• controllable generation

Innovation

• dynamic guidance
• reinforcement learning
• diffusion language models
• classifier-free guidance
• controllable generation