Learning When Not to Attend Globally

2025-12-27

AI Summary

To address the computational redundancy and inefficiency of global attention in large language models (LLMs), this paper proposes All-or-Here Attention (AHA), a dynamic attention mechanism that determines—per token—whether global contextual modeling is necessary; otherwise, it resorts to local sliding-window attention. Our key contributions are: (i) the first empirical identification of a long-tail distribution in contextual dependency lengths across tokens; and (ii) the design of head-wise binary routing gates that enable fine-grained, on-demand switching between full attention and windowed attention (window size = 256). Experiments demonstrate that AHA replaces 93% of full-attention computations with no performance degradation, maintains state-of-the-art results across multiple long-context benchmarks (e.g., Needle-in-a-Haystack, LongBench), and significantly improves inference efficiency and scalability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

When reading books, humans focus primarily on the current page, flipping back to recap prior context only when necessary. Similarly, we demonstrate that Large Language Models (LLMs) can learn to dynamically determine when to attend to global context. We propose All-or-Here Attention (AHA), which utilizes a binary router per attention head to dynamically toggle between full attention and local sliding window attention for each token. Our results indicate that with a window size of 256 tokens, up to 93% of the original full attention operations can be replaced by sliding window attention without performance loss. Furthermore, by evaluating AHA across various window sizes, we identify a long-tail distribution in context dependency, where the necessity for full attention decays rapidly as the local window expands. By decoupling local processing from global access, AHA reveals that full attention is largely redundant, and that efficient inference requires only on-demand access to the global context.

Problem

• Dynamic global context attention in LLMs
• Replace full attention with sliding window
• Reduce redundancy in attention operations

Innovation

• Dynamic binary router per attention head
• Toggle between full and local sliding window attention
• On-demand global context access reduces redundancy