Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension

2026-08-10

AI Summary

This study investigates how subtle architectural design choices in dense Transformers affect their ability to scale to long-context tasks. Through controlled experiments, the authors train 26 distinct 7B-parameter models (OlmPool) under fixed data, tokenizer, and context-extension strategies, systematically evaluating the combined impact of normalization schemes, grouped-query attention (GQA), pretraining context length, and sliding-window attention. They find that combinations of three or four suboptimal design choices can degrade downstream long-context performance by up to 47%, a drop not predictable from short-context metrics. Crucially, such degradation can be reliably detected early in pretraining through patterns in attention distributions. Notably, certain novel architectures examined in this work substantially outperform Llama 3 in long-context scaling.

Abstract

One might imagine that architectural variations within the dense transformer paradigm have a limited effect on accuracy. However, we demonstrate that this is not the case in the long context setting. Specifically, we show that a set of four minor architectural decisions --- all made by at least one of the Olmo, Llama, and Qwen dense model families --- have a compoundingly negative effect on long context extensibility. Any one of these choices alone has a minor impact on long context performance, but combining three or more can drop the performance downstream by up to 47%. Furthermore, these differences are not detectable from short-context loss or validation datasets. We show that much of the variation in long context ability across model families is driven by these architectural features and detectable from applying context extension early in pretraining. We demonstrate this with controlled ablations that hold data, tokenizer, and extension recipe fixed while varying normalization, GQA, pretraining context length, and sliding window attention. After over 170,000 GPU hours of training, we release the resulting set of models as OlmPool, a set of 26 comparable 7B models with checkpoints before and after long-context extension. This pool includes several architectures that outperform the Llama 3 architecture on long context extensibility. In an analysis of our ablation models, we identify patterns in attention sink behavior and attention distributions across context that are attributable to specific architectural differences.

Problem

• long context extension
• architectural choices
• dense transformer
• attention mechanisms
• model extensibility

Innovation

• long-context extension
• architectural ablation
• attention sink
• grouped-query attention
• transformer normalization