Attention Sinks in Diffusion Language Models

2025-10-17

AI Summary

This work systematically investigates the “attention sinking” phenomenon in diffusion language models (DLMs), revealing a fundamental distinction from autoregressive models (ARMs): in DLMs, the sink location dynamically migrates during generation rather than remaining fixed at the sequence end. Leveraging a Transformer encoder with bidirectional attention and masked diffusion training, we conduct attention visualization and masking ablation experiments. Results show that DLMs exhibit remarkable robustness to masking of the sunk region—removing sunk attention incurs only marginal performance degradation (<0.5%), indicating greater flexibility and redundancy in attention allocation. To our knowledge, this is the first study to characterize the dynamic attention sinking mechanism in DLMs. Our findings provide a novel perspective on the generation robustness of diffusion-based language modeling and challenge the conventional ARM assumption that attention concentrates within a local context window.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Masked Diffusion Language Models (DLMs) have recently emerged as a promising alternative to traditional Autoregressive Models (ARMs). DLMs employ transformer encoders with bidirectional attention, enabling parallel token generation while maintaining competitive performance. Although their efficiency and effectiveness have been extensively studied, the internal mechanisms that govern DLMs remain largely unexplored. In this work, we conduct an empirical analysis of DLM attention patterns, focusing on the attention sinking phenomenon, an effect previously observed in various transformer-based architectures. Our findings reveal that DLMs also exhibit attention sinks, but with distinct characteristics. First, unlike in ARMs, the sink positions in DLMs tend to shift throughout the generation process, displaying a dynamic behaviour. Second, while ARMs are highly sensitive to the removal of attention sinks, DLMs remain robust: masking sinks leads to only a minor degradation in performance. These results provide new insights into the inner workings of diffusion-based language models and highlight fundamental differences in how they allocate and utilize attention compared to autoregressive models.

Problem

• Analyzing attention sinking phenomenon in diffusion language models
• Comparing dynamic sink patterns between diffusion and autoregressive models
• Investigating robustness of diffusion models when masking attention sinks

Innovation

• DLMs use bidirectional attention for parallel generation
• Dynamic attention sinks shift during generation process
• DLMs remain robust when attention sinks are masked