VoidPadding: Let [VOID] Handle Padding in Masked Diffusion Language Models so that [EOS] Can Focus on Semantic Termination

2026-06-16

AI Summary

This work addresses a critical limitation in existing masked diffusion language models, where the end-of-sequence token ([EOS]) concurrently serves both semantic termination and padding roles during instruction tuning, often leading to [EOS] overflow during large-block decoding. To resolve this, the authors propose VoidPadding, a novel approach that decouples these dual functions by introducing a dedicated padding token, [VOID], thereby allowing [EOS] to exclusively signal semantic completion. This design enables an early-stopping mechanism during inference and supports adaptive response canvas expansion. Experiments on Dream-7B-Instruct demonstrate that VoidPadding improves average performance by 17.84 points over the original model and by 6.95 points over RainbowPadding across four tasks, while reducing the average number of function evaluations (NFE) by 55.7%, substantially enhancing both generation stability and computational efficiency.

Abstract

MDLMs generate text by denoising a preallocated masked response canvas, making response-length modeling central to instruction tuning. Existing MDLMs often inherit the autoregressive convention of using repeated \texttt{[EOS]} tokens for padding during instruction tuning, giving \texttt{[EOS]} a dual role as both a semantic terminator and a padding token. We show that this dual role is a root cause of \texttt{[EOS]} overflow under large-block decoding. To decouple these roles, we propose VoidPadding, which introduces \texttt{[VOID]} for padding and reserves \texttt{[EOS]} for termination. During inference, the learned \texttt{[EOS]} signal enables early stopping, while the learned \texttt{[VOID]} signal guides adaptive response canvas expansion. On Dream-7B-Instruct, VoidPadding improves the block-size-averaged four-task mean across mathematical reasoning and code generation benchmarks by \(+17.84\) points over the original model and \(+6.95\) points over RainbowPadding, while reducing decoding NFE by 55.7\% on average. Code is available at https://github.com/Haru-LCY/VoidPadding.

Problem

• Masked Diffusion Language Models
• EOS token
• padding
• semantic termination
• decoding overflow

Innovation

• VoidPadding
• Masked Diffusion Language Models
• EOS overflow
• adaptive canvas expansion
• early stopping