ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation

2026-06-16

AI Summary

Existing hybrid attention architectures rely on handcrafted rules to allocate full and sliding-window attention, making them ill-suited for user-specified sparsity targets and lacking fine-grained interpretability of attention behavior. This work proposes ConSA, a framework that, for the first time, enables controllable sparsity in attention allocation at either the layer or key-value (KV) head granularity. ConSA learns binary masks via L0 regularization to automatically select attention types and enforces strict sparsity constraints through an augmented Lagrangian method. Experiments demonstrate that ConSA consistently outperforms rule-based baselines on both 0.6B and 1.7B models, with KV-head-level allocation significantly surpassing layer-level allocation. Moreover, the learned attention patterns remain stable across varying model scales and sparsity levels, overcoming the limitations of conventional uniformly interleaved allocation schemes.

Abstract

Hybrid architectures combining full attention (FA) and sliding-window attention (SWA) are a promising paradigm for efficient LLM inference. However, existing methods typically rely on hand-crafted rules or simple post-hoc heuristics for FA/SWA allocation and offer limited analysis of the attention behaviors underlying these designs. We propose Controllable Sparsity in Hybrid Attention (ConSA), a framework that learns optimal FA/SWA assignment under a user-specified sparsity target. ConSA employs L0 regularization to learn binary masks selecting between FA and SWA for each attention unit, while an augmented Lagrangian constraint enforces the target sparsity at either layer or KV-head granularity. We evaluate ConSA on two LLMs at the 0.6B and 1.7B scales. Learned allocations consistently outperform rule-based baselines, with KV-head-wise allocation yielding clear gains over layer-wise allocation. The learned patterns place SWA in the bottom layers and concentrate FA into contiguous middle-layer blocks, diverging from evenly interleaved patterns in rule-based methods. This structure persists across model scales, sparsity levels, and allocation granularities, revealing a fine-grained spectrum of intrinsic attention behaviors that underlies the learned allocation.

Problem

• hybrid attention
• sparsity control
• attention allocation
• large language models
• efficient inference

Innovation

• controllable sparsity
• hybrid attention
• learnable allocation
• L0 regularization
• augmented Lagrangian