Swordsman: Entropy-Driven Adaptive Block Partition for Efficient Diffusion Language Models

2026-02-04

AI Summary

This work addresses a critical limitation in existing block decoding methods for diffusion language models, which rely on fixed chunking strategies that often disrupt semantic or syntactic coherence, thereby degrading both generation quality and efficiency. To overcome this, the authors propose Swordsman, a novel framework that introduces the entropy reduction hypothesis into diffusion language modeling for the first time. By dynamically monitoring entropy changes during decoding, Swordsman adaptively identifies semantic boundaries to enable training-free block segmentation. Furthermore, it incorporates real-time mask state awareness to adjust unmasking thresholds dynamically and integrates a KV Cache acceleration mechanism. Experimental results demonstrate that this approach significantly enhances inference speed and generation quality across multiple benchmarks, achieving state-of-the-art performance.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Block-wise decoding effectively improves the inference speed and quality in diffusion language models (DLMs) by combining inter-block sequential denoising and intra-block parallel unmasking. However, existing block-wise decoding methods typically partition blocks in a rigid and fixed manner, which inevitably fragments complete semantic or syntactic constituents, leading to suboptimal performance. Inspired by the entropy reduction hypothesis (ERH), we recognize that constituent boundaries offer greater opportunities for uncertainty reduction, which motivates us to employ entropy analysis for identifying constituent boundaries. Therefore, we propose Swordsman, an entropy-driven adaptive block-wise decoding framework for DLMs. Swordsman adaptively partitions blocks by identifying entropy shifts between adjacent tokens to better align with semantic or syntactic constituent boundaries. In addition, Swordsman dynamically adjusts unmasking thresholds conditioned on the real-time unmasking status within a block, further improving both efficiency and stability. As a training-free framework, supported by KV Cache, Swordsman demonstrates state-of-the-art performance across extensive evaluations.

Problem

• block-wise decoding
• diffusion language models
• constituent boundaries
• entropy
• adaptive partition

Innovation

• entropy-driven
• adaptive block partition
• diffusion language models
• block-wise decoding
• constituent boundary detection