In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Tasks

2026-02-19

AI Summary

This study investigates the performance discrepancy between linear and quadratic attention mechanisms in in-context learning (ICL) for linear regression tasks. Building upon the standard Transformer architecture, we conduct a systematic comparison under controlled conditions to evaluate their relative efficacy in learning quality, convergence behavior, and generalization capability, while also examining the influence of model depth on ICL performance. Using mean squared error (MSE) as the primary evaluation metric, our experiments reveal that although both attention mechanisms exhibit comparable overall performance in ICL, linear attention encounters specific performance bottlenecks under certain configurations. Furthermore, model depth is found to significantly modulate ICL effectiveness. These findings elucidate both the advantages and limitations of linear attention in ICL settings, offering empirical insights to guide the design of efficient attention mechanisms.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent work has demonstrated that transformers and linear attention models can perform in-context learning (ICL) on simple function classes, such as linear regression. In this paper, we empirically study how these two attention mechanisms differ in their ICL behavior on the canonical linear-regression task of Garg et al. We evaluate learning quality (MSE), convergence, and generalization behavior of each architecture. We also analyze how increasing model depth affects ICL performance. Our results illustrate both the similarities and limitations of linear attention relative to quadratic attention in this setting.

Problem

• In-Context Learning
• Linear Attention
• Quadratic Attention
• Regression Tasks
• Transformers

Innovation

• In-Context Learning
• Linear Attention
• Quadratic Attention
• Empirical Study
• Regression Tasks