Customizing the Inductive Biases of Softmax Attention using Structured Matrices

2025-09-09

AI Summary

Standard attention suffers from information loss under high-dimensional inputs due to low-rank projections and lacks explicit distance-aware bias for neighboring tokens. To address this, we propose a novel attention scoring function that— for the first time—integrates Block Tensor-Train (BTT) and Multi-Level Low-Rank (MLR) structured matrices into the attention mechanism. This design enables full-rank modeling and explicit positional bias encoding while preserving near-linear computational complexity. Crucially, it unifies high-rank expressivity with inductive biases—without increasing parameter count or compromising inference efficiency. Experiments across contextual regression, language modeling, and long-horizon time-series forecasting demonstrate consistent and significant improvements over standard attention and sliding-window variants, achieving superior generalization and scalability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The core component of attention is the scoring function, which transforms the inputs into low-dimensional queries and keys and takes the dot product of each pair. While the low-dimensional projection improves efficiency, it causes information loss for certain tasks that have intrinsically high-dimensional inputs. Additionally, attention uses the same scoring function for all input pairs, without imposing a distance-dependent compute bias for neighboring tokens in the sequence. In this work, we address these shortcomings by proposing new scoring functions based on computationally efficient structured matrices with high ranks, including Block Tensor-Train (BTT) and Multi-Level Low Rank (MLR) matrices. On in-context regression tasks with high-dimensional inputs, our proposed scoring functions outperform standard attention for any fixed compute budget. On language modeling, a task that exhibits locality patterns, our MLR-based attention method achieves improved scaling laws compared to both standard attention and variants of sliding window attention. Additionally, we show that both BTT and MLR fall under a broader family of efficient structured matrices capable of encoding either full-rank or distance-dependent compute biases, thereby addressing significant shortcomings of standard attention. Finally, we show that MLR attention has promising results for long-range time-series forecasting.

Problem

• Addressing information loss in low-dimensional attention projections
• Introducing distance-dependent compute biases for sequence modeling
• Enhancing attention with efficient high-rank structured matrices

Innovation

• Structured matrices for high-rank scoring functions
• Block Tensor-Train and Multi-Level Low Rank matrices
• Encoding full-rank or distance-dependent compute biases