Diffusion LMs Can Approximate Optimal Infilling Lengths Implicitly

2026-01-31

AI Summary

This work addresses the limitation of diffusion language models in infilling tasks, where predefined sequence lengths hinder adaptive determination of the optimal fill length. The authors identify two phenomena in the confidence of the first-step denoising: an “Oracle Peak” and a “Length Bias.” Leveraging these insights, they propose CAL (Confidence-based Adaptive Length), a training-free method that efficiently searches for an approximately optimal length prior to formal decoding through confidence analysis and bias calibration. CAL is the first approach to achieve implicit, training-free length adaptation in diffusion-based infilling. On code infilling benchmarks, it improves Pass@1 by up to 47.7% over fixed-length and chat-based adaptive baselines; on textual infilling, it yields relative gains of 8.5% in BLEU-2 and 9.9% in ROUGE-L.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Diffusion language models (DLMs) provide a bidirectional generation framework naturally suited for infilling, yet their performance is constrained by the pre-specified infilling length. In this paper, we reveal that DLMs possess an inherent ability to discover the correct infilling length. We identify two key statistical phenomena in the first-step denoising confidence: a local \textit{Oracle Peak} that emerges near the ground-truth length and a systematic \textit{Length Bias} that often obscures this signal. By leveraging this signal and calibrating the bias, our training-free method \textbf{CAL} (\textbf{C}alibrated \textbf{A}daptive \textbf{L}ength) enables DLMs to approximate the optimal length through an efficient search before formal decoding. Empirical evaluations demonstrate that CAL improves Pass@1 by up to 47.7\% over fixed-length baselines and 40.5\% over chat-based adaptive methods in code infilling, while boosting BLEU-2 and ROUGE-L by up to 8.5\% and 9.9\% in text infilling. These results demonstrate that CAL paves the way for robust DLM infilling without requiring any specialized training. Code is available at https://github.com/NiuHechang/Calibrated_Adaptive_Length.

Problem

• Diffusion Language Models
• Infilling Length
• Length Bias
• Oracle Peak
• Text Infilling

Innovation

• Diffusion Language Models
• Infilling Length Adaptation
• Training-Free Calibration
• Oracle Peak
• Length Bias