Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch

2026-07-03

AI Summary

Existing theories of in-context learning typically assume that demonstration examples in prompts are independent and identically distributed, thereby overlooking the pervasive temporal correlations present in real-world sequences. This work develops a theoretically tractable model based on linear attention, integrating a linear regression theory sandbox with an actual Transformer architecture to systematically investigate how temporal dependencies within prompts affect in-context learning. We reveal for the first time that such temporal correlations induce an “effective context length,” rendering correlated prompts equivalent to shorter i.i.d. ones. Moreover, we find that when queries are temporally aligned with the context, Softmax attention substantially outperforms linear attention, highlighting the critical importance of aligning attention mechanisms with task-specific structural properties.

Abstract

Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has served as a useful sandbox for ICL theory, but existing work has largely focused on prompts with independent examples. In this work, we extend this setting to sequentially correlated data, a basic feature of real sequences. We present a solvable model based on linear attention and test our predictions on realistic transformer architectures. We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length: correlated prompts behave like shorter i.i.d. prompts. Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention. These results suggest that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.

Problem

• in-context learning
• sequential correlations
• effective context length
• architectural mismatch
• attention mechanisms

Innovation

• in-context learning
• sequential correlations
• effective context length
• attention architecture
• linear attention