On the locality bias and results in the Long Range Arena

2025-01-24

AI Summary

This study reveals a strong locality bias in the Long-Range Arena (LRA) benchmark, demonstrating that it primarily evaluates short-range rather than true long-range dependencies. Method: Through systematic analysis, we show that Transformer performance gains on LRA stem largely from improved positional encoding (e.g., RoPE) and denoising pretraining—not architectural advances; similarly, SSMs and MEGA excel due to task-specific inductive biases, not superior long-range modeling. We empirically identify LRA’s structural flaws and propose three key improvements: denoising pretraining, parameterized convolutions replacing fixed SSM kernels, and data-efficient training. Contribution/Results: These enhancements enable Transformers to achieve state-of-the-art performance on LRA, while full-parameter convolutional SSMs match this performance. Our findings expose fundamental limitations of current long-sequence evaluation paradigms and call for benchmarks explicitly designed to rigorously test genuine long-range dependency modeling—providing both theoretical insight and practical guidance for future long-context model assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The Long Range Arena (LRA) benchmark was designed to evaluate the performance of Transformer improvements and alternatives in long-range dependency modeling tasks. The Transformer and its main variants performed poorly on this benchmark, and a new series of architectures such as State Space Models (SSMs) gained some traction, greatly outperforming Transformers in the LRA. Recent work has shown that with a denoising pre-training phase, Transformers can achieve competitive results in the LRA with these new architectures. In this work, we discuss and explain the superiority of architectures such as MEGA and SSMs in the Long Range Arena, as well as the recent improvement in the results of Transformers, pointing to the positional and local nature of the tasks. We show that while the LRA is a benchmark for long-range dependency modeling, in reality most of the performance comes from short-range dependencies. Using training techniques to mitigate data inefficiency, Transformers are able to reach state-of-the-art performance with proper positional encoding. In addition, with the same techniques, we were able to remove all restrictions from SSM convolutional kernels and learn fully parameterized convolutions without decreasing performance, suggesting that the design choices behind SSMs simply added inductive biases and learning efficiency for these particular tasks. Our insights indicate that LRA results should be interpreted with caution and call for a redesign of the benchmark.

Problem

• Position Bias
• Transformer Models
• Long-Range Arena

Innovation

• Transformer Optimization
• State Space Models
• Long Range Arena