Overflow Prevention Enhances Long-Context Recurrent LLMs

2025-05-12

AI Summary

Recurrent language models (RLMs) suffer from fixed-size recurrent memory, leading to overflow and insufficient long-range information utilization in long-context scenarios. Method: We propose a relevance-driven chunked inference method that employs a lightweight attention mechanism to dynamically identify and select the most critical input segments, coupled with recurrent state truncation and reinitialization for efficient localized processing. Contribution/Results: This approach challenges the conventional assumption that RLMs inherently require long-term memory, and for the first time reveals the fundamental inefficiency in recurrent memory utilization. Evaluated on LongBench, our method achieves average improvements of 14%–51%; its v2 variant matches the performance of transformer-based models of comparable scale, establishing a new state-of-the-art for recurrent architectures in long-context reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

A recent trend in LLMs is developing recurrent sub-quadratic models that improve long-context processing efficiency. We investigate leading large long-context models, focusing on how their fixed-size recurrent memory affects their performance. Our experiments reveal that, even when these models are trained for extended contexts, their use of long contexts remains underutilized. Specifically, we demonstrate that a chunk-based inference procedure, which identifies and processes only the most relevant portion of the input can mitigate recurrent memory failures and be effective for many long-context tasks: On LongBench, our method improves the overall performance of Falcon3-Mamba-Inst-7B by 14%, Falcon-Mamba-Inst-7B by 28%, RecurrentGemma-IT-9B by 50%, and RWKV6-Finch-7B by 51%. Surprisingly, this simple approach also leads to state-of-the-art results in the challenging LongBench v2 benchmark, showing competitive performance with equivalent size Transformers. Furthermore, our findings raise questions about whether recurrent models genuinely exploit long-range dependencies, as our single-chunk strategy delivers stronger performance - even in tasks that presumably require cross-context relations.

Problem

• Investigates recurrent memory impact on long-context LLM performance
• Proposes chunk-based inference to mitigate recurrent memory failures
• Questions if recurrent models truly exploit long-range dependencies

Innovation

• Chunk-based inference for relevant input processing
• Overflow prevention in recurrent memory models
• Single-chunk strategy enhances long-context performance