Who Should Lead Decoding Now? Tracking Reliable Trajectories for Ensembling Masked Diffusion Language Models

2026-06-15

AI Summary

This work addresses the inefficiency of knowledge fusion in ensembles of multi-masked diffusion language models (MDLMs) by proposing the TIE framework. It reveals, for the first time, the dynamic confidence patterns at answer-relevant positions during MDLM decoding and leverages this insight to design an iterative ensemble mechanism based on confidence trajectory tracking. By integrating partial denoising sequence relaying with multi-model collaborative decoding, TIE dynamically selects the optimal model at each denoising stage to achieve complementary strengths across phases. Experimental results demonstrate that TIE significantly enhances generation quality across diverse reasoning tasks, validating its effectiveness and practicality in MDLM ensembling.

Abstract

Masked Diffusion Language Models (MDLMs) have emerged as a distinct paradigm for sequence generation. As MDLMs become diverse in capabilities and knowledge coverage, an important question is how to combine their knowledge. Toward this, we first investigate the unique decoding dynamics of MDLMs. We find that successful generations exhibit stable confidence dynamics over answer-relevant positions, while unreliable trajectories can often be corrected by injecting promising intermediate states from other models. Guided by this observation, we propose $\textbf{TIE}$ ($\textbf{T}$rajectory-based $\textbf{I}$terative $\textbf{E}$nsembling), a knowledge fusion framework in which MDLMs iteratively identify reliable decoding trajectories and relay them across models. TIE tracks confidence dynamics over answer-relevant positions to determine which model currently follows a more reliable trajectory and selectively transfers partially denoised sequences across models. As the model on the more promising trajectory often changes across denoising steps, TIE allows different models to contribute complementary strengths at different stages of generation. Strong performance across diverse reasoning tasks, along with our analyses, suggests that TIE offers a practical approach to the underexplored problem of MDLM ensembling.

Problem

• Masked Diffusion Language Models
• Ensembling
• Decoding Trajectories
• Knowledge Fusion
• Sequence Generation

Innovation

• Masked Diffusion Language Models
• Ensembling
• Decoding Trajectory
• Confidence Dynamics
• Iterative Knowledge Fusion