Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding

2026-08-31

AI Summary

为解决长上下文解码中的内存带宽瓶颈和计算效率问题，提出了一种硬件-算法协同设计框架FFD，通过内容感知扫描和动态筛选策略提高解码速度和稀疏性。

Abstract

The development of long-context Large Language Models (LLMs) is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding. To overcome the inherent trade-offs between the memory overhead of metadata-based metrics and the computational inefficiency of adaptive selection strategies, we present Faster Flash Decoding (FFD), a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding. FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning via low-bit quantization. Furthermore, we introduce the top-delta strategy, which dynamically filters blocks to achieve distribution-adaptive sparsity without global synchronization. Offering a training-free and plug-and-play solution, FFD also enables the reuse of scanning results for computation, achieving up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement. Empirical validation on RULER and LongBench confirms that FFD maintains model accuracy while delivering high-ratio sparsity, with code available at https://github.com/qluoluo/faster-flash-decoding

Problem

• Long-context Large Language Models
• memory bandwidth bottleneck
• attention mechanism
• decoding

Innovation

• Faster Flash Decoding
• content-aware scanning
• low-bit quantization
• top-delta strategy
• distribution-adaptive sparsity