Tapered Language Models

2026-06-22

AI Summary

This work proposes a depth-aware tapered parameter allocation strategy that addresses the inefficiency of uniform layer-wise parameter distribution in conventional language models, which overlooks the substantial heterogeneity in the contribution of different network depths to model outputs. By monotonically decreasing the MLP width along the network depth under a fixed parameter budget, the method systematically leverages inter-layer contribution disparities to establish an architecture-agnostic design principle. Evaluated across four distinct architectures—Transformer, Gated Attention, Hope-attention, and Titans—and three model scales, the proposed approach consistently outperforms uniform-width baselines, yielding significant improvements in both perplexity and downstream task performance without incurring additional computational overhead.

Abstract

Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly across depth. This is a default inherited from the original transformer and largely unchanged since, yet a growing body of evidence suggests that layers contribute non-uniformly to the final output, with later layers refining the residual stream rather than transforming it. We ask whether parameter capacity should reflect this asymmetry. Our controlled experiment shows that, under a fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation hurts. Building on this result, we introduce Tapered Language Models (TLMs), an architectural principle in which a parameter-bearing component is monotonically tapered across depth under a fixed total budget. MLPs are the natural site for this instantiation: they dominate parameter count across all modern LM families and expose width as a single, clean axis of variation. Across three model scales and four architectures (Transformer, Gated Attention, Hope-attention, and Titans), tapering MLP width via a smooth cosine schedule consistently improves perplexity and downstream benchmark performance over uniform baselines, at no additional parameter or compute cost. These findings establish depth-aware capacity allocation as a simple, architecture-agnostic axis of language model design, a free lever hidden in plain sight.

Problem

• language models
• parameter allocation
• model depth
• capacity distribution
• uniform architecture

Innovation

• Tapered Language Models
• depth-aware capacity allocation
• non-uniform layer width
• parameter budget
• MLP tapering