MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers

2026-06-29

AI Summary

Traditional attention mechanisms struggle to scale to long-context scenarios due to their quadratic computational complexity, while existing sparse attention methods often compromise long-range dependency modeling because of structural constraints. This work proposes MATCH, a novel framework that seamlessly integrates dynamic context retrieval into sparse attention for the first time. By leveraging efficient vector search to retrieve critical distant information in real time and dynamically fusing it with sparse attention, MATCH significantly enhances precise recall capabilities in long-context tasks without sacrificing computational efficiency. Experimental results demonstrate that MATCH consistently outperforms current sparse attention models on both synthetic and real-world natural language benchmarks, confirming its effectiveness and generality in strengthening long-range modeling capacity.

Abstract

The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particularly in long-context scenarios. To improve efficiency, existing approaches often enforce rigid structural constraints such as local attention windows. However, these strategies typically lead to substantial performance degradation on tasks requiring precise long-range recall. In this work, we propose MATCH, a scalable and efficient framework that augments sparsified attention mechanisms with dynamically integrated in-context information through an efficient retrieval system. Empirical results show that MATCH significantly improves the performance of sparse-attention models on both synthetic and real-world natural-language tasks. These findings highlight the versatility of MATCH as a general approach for enhancing in-context retrieval capabilities while maintaining the efficiency benefits of sparse attention architectures.

Problem

• long-context
• attention mechanism
• computational cost
• sparse attention
• in-context retrieval

Innovation

• in-context retrieval
• sparse attention
• long-context transformers
• efficient retrieval
• attention modulation