Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models

2026-02-02

AI Summary

Existing masked diffusion language models rely on confidence-based greedy sampling, which often yields limited diversity and heightened sensitivity to noise. To address this, this work proposes a self-rewarding Sequential Monte Carlo (SMC) approach that propagates multiple diffusion particles in parallel during inference. For the first time, trajectory-level confidence is leveraged as an intrinsic self-reward signal to dynamically weight and resample particles, thereby guiding the generation toward globally coherent, high-confidence, and high-quality text. Notably, the method requires neither additional training nor external reward signals. Evaluated across multiple masked diffusion models and benchmarks, it consistently achieves significant improvements in both generation quality and diversity, effectively translating the inherent parallelism of diffusion inference into tangible performance gains.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

This work presents self-rewarding sequential Monte Carlo (SMC), an inference-time scaling algorithm enabling effective sampling of masked diffusion language models (MDLMs). Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy, where only tokens with the highest prediction confidence are preserved at each step. This restricts the generation to a noise-sensitive, greedy decoding paradigm, resulting in an inevitable collapse in the diversity of possible paths. We address this problem by launching multiple interacting diffusion processes in parallel, referred to as particles, for trajectory exploration. Importantly, we introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights. During sampling, particles are iteratively weighted and resampled to systematically steer generation towards globally confident, high-quality samples. Our self-rewarding SMC is verified on various masked diffusion language models and benchmarks, achieving significant improvement without extra training or reward guidance, while effectively converting parallel inference capacity into improved sampling quality. Our code is available at https://github.com/Algolzw/self-rewarding-smc.

Problem

• masked diffusion language models
• sampling diversity collapse
• confidence-based sampling
• greedy decoding
• trajectory exploration

Innovation

• self-rewarding SMC
• masked diffusion language models
• trajectory-level confidence
• parallel particle sampling
• inference-time scaling