Don't Pay Attention

2025-06-12

AI Summary

Transformers face fundamental limitations in modeling ultra-long sequences due to fixed context windows and quadratic computational complexity; RNNs, while linear in sequence length, suffer from poor parallelizability. This paper introduces Avey, the first architecture to entirely abandon both attention and recurrence paradigms. Instead, it synergistically combines a token-rank sorter with a lightweight autoregressive neural processor to perform position-agnostic critical token selection and modeling—thereby fully decoupling context width from sequence length. This design enables unbounded-length long-range dependency modeling with near-linear time complexity. Experiments demonstrate that Avey matches Transformer performance on standard short-context NLP tasks, while substantially outperforming state-of-the-art models on long-range benchmarks—including PG19 and the Long Range Arena—validating both the efficacy and scalability of this novel architectural paradigm.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The Transformer has become the de facto standard for large language models and a wide range of downstream tasks across various domains. Despite its numerous advantages like inherent training parallelism, the Transformer still faces key challenges due to its inability to effectively process sequences beyond a fixed context window and the quadratic complexity of its attention mechanism. These challenges have renewed interest in RNN-like architectures, which offer linear scaling with sequence length and improved handling of long-range dependencies, albeit with limited parallelism due to their inherently recurrent nature. In this paper, we propose Avey, a new neural foundational architecture that breaks away from both attention and recurrence. Avey comprises a ranker and an autoregressive neural processor, which collaboratively identify and contextualize only the most relevant tokens for any given token, regardless of their positions in the sequence. Specifically, Avey decouples sequence length from context width, thus enabling effective processing of arbitrarily long sequences. Experimental results show that Avey compares favorably to the Transformer across a variety of standard short-range NLP benchmarks, while notably excelling at capturing long-range dependencies.

Problem

• Overcoming fixed context window limits in Transformers
• Reducing quadratic complexity of attention mechanisms
• Improving long-range dependency handling without recurrence

Innovation

• Avey replaces attention and recurrence mechanisms
• Decouples sequence length from context width
• Ranker and neural processor identify relevant tokens