Attention in Diffusion Model: A Survey

2025-04-01

AI Summary

Attention mechanisms in diffusion models lack systematic analysis regarding their functional roles, design principles, and cross-modal/task generalizability. Method: We propose the first unified taxonomy for attention modifications in diffusion models, categorizing improvements by architectural component—e.g., U-Net backbone, cross-attention, spatial/channel-wise attention—and integrating multi-dimensional analysis: architectural characterization, modality-aware comparison, performance attribution, and limitation diagnosis. Contribution/Results: Our analysis reveals distinct contribution pathways of attention to generation quality, training stability, sampling efficiency, and controllable editing. We identify critical bottlenecks: poor scalability, high computational redundancy, and weak theoretical interpretability. The framework provides a structured design guide for attention-augmented diffusion models and motivates future directions—including attention sparsification, modular co-optimization, and interpretable attention modeling—to advance both efficacy and understanding.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Attention mechanisms have become a foundational component in diffusion models, significantly influencing their capacity across a wide range of generative and discriminative tasks. This paper presents a comprehensive survey of attention within diffusion models, systematically analysing its roles, design patterns, and operations across different modalities and tasks. We propose a unified taxonomy that categorises attention-related modifications into parts according to the structural components they affect, offering a clear lens through which to understand their functional diversity. In addition to reviewing architectural innovations, we examine how attention mechanisms contribute to performance improvements in diverse applications. We also identify current limitations and underexplored areas, and outline potential directions for future research. Our study provides valuable insights into the evolving landscape of diffusion models, with a particular focus on the integrative and ubiquitous role of attention.

Problem

• Survey attention mechanisms in diffusion models
• Analyze roles and designs of attention
• Identify limitations and future research directions

Innovation

• Survey attention mechanisms in diffusion models
• Propose unified taxonomy for attention modifications
• Analyze performance impact across diverse applications