AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation

2026-04-20

AI Summary

This work addresses the high inference latency of video diffusion Transformers caused by the quadratic complexity of attention mechanisms. Existing sparse attention methods often compromise generation quality by either ignoring semantic similarity or failing to adapt to the heterogeneous token distributions across layers. To overcome these limitations, the authors propose AdaCluster, a training-free adaptive clustering framework that employs a dual-path clustering strategy—preserving angular similarity for queries and Euclidean similarity for keys—to dynamically determine cluster counts, adaptively set thresholds, and efficiently select salient clusters. This approach effectively compresses attention while maintaining semantic fidelity. Evaluated on CogVideoX-2B, HunyuanVideo, and Wan-2.1, AdaCluster achieves speedups of 1.67× to 4.31× on a single A40 GPU with negligible degradation in generation quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Clustering
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity. Existing sparse attention methods either overlook semantic similarity or fail to adapt to heterogeneous token distributions across layers, leading to model performance degradation. We propose AdaCluster, a training-free adaptive clustering framework that accelerates the generation of DiTs while preserving accuracy. AdaCluster applies an angle-similarity-preserving clustering method to query vectors for higher compression, and designs a euclidean-similarity-preserving clustering method for keys, covering cluster number assignment, threshold-wise adaptive clustering, and efficient critical cluster selection. Experiments on CogVideoX-2B, HunyuanVideo, and Wan-2.1 on one A40 GPU demonstrate up to 1.67-4.31x speedup with negligible quality degradation.

Problem

• sparse attention
• video diffusion transformers
• attention complexity
• token distribution
• inference latency

Innovation

• adaptive clustering
• sparse attention
• video diffusion transformers
• angle-similarity-preserving
• euclidean-similarity-preserving