CEDAR: Error-Bounded Residual Routing for Efficient Long-Context Attention

2026-09-07

AI Summary

CEDAR通过粗到细的方法和错误感知动态注意力路由解决长上下文注意力效率问题，保持全局覆盖同时减少重建误差。

Abstract

Post-hoc sparse attention accelerates long-context prefill by routing each query to a small set of token-level interactions. Hard selection, however, assigns zero probability to every omitted chunk: a routing miss cannot be recovered, and a fixed expansion budget spends the same work on easy and ambiguous queries. We introduce Coarse-to-fine Error-aware Dynamic Attention Routing (CEDAR), a coarse-to-fine method that keeps the language model frozen while preserving global coverage. Each semantic chunk contributes a cheap key--value summary to a residual attention path; chunks with high estimated approximation error are then expanded to exact token attention. Exact and summarized contributions are combined in a single softmax normalization, so refinement replaces, rather than duplicates, coarse evidence. We derive an output-error bound governed by within-chunk key/value dispersion and use it to allocate a variable refinement budget. A controlled clustered-attention study shows that residual summaries reduce reconstruction error by more than 98% relative to hard dropping at equal exact-chunk budgets. Experiments on long-context benchmarks demonstrate that CEDAR recovers most of the quality lost by hard sparse routing while maintaining approximately $3\times$ kernel speedup at 128K context.

Problem

• sparse attention
• long-context prefill
• hard selection
• routing miss
• expansion budget

Innovation

• Error-aware Dynamic Attention Routing
• Residual Summaries
• Global Coverage Preservation
• Variable Refinement Budget