Full-bandwidth transformer

2026-08-09

AI Summary

This work addresses the limitation of conventional autoregressive Transformers, which discard top-layer hidden states during decoding by only feeding back sampled tokens, thereby underutilizing deep computational representations. To overcome this, the authors propose a latent feedback mechanism that, at each decoding step, fuses the previous step’s top-layer hidden state with the token embedding via a gated linear unit and re-injects it into the model, effectively widening the vertical information pathway without altering the standard Transformer architecture. Combined with KV caching and a scheduled multi-phase pretraining strategy, the method achieves substantial improvements in validation loss, 5-shot language modeling, mathematical and code generation, and instruction fine-tuning—while incurring negligible inference overhead. Notably, a 1B-parameter model trained on 400B tokens matches or nearly matches the performance of standard models trained on 1.5× more data and produces shorter yet accurate reasoning trajectories.

Abstract

Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal access to the past, but the vertical feedback channel between decoding steps remains narrow: only the sampled token returns to the bottom of the stack, while the top-layer hidden state is discarded. We introduce the \emph{full-bandwidth transformer}, which widens this channel with \emph{latent feedback}: at each decoding step, the previous top-layer hidden state is fused with the sampled token embedding through a gated linear unit and fed back as the next input. Latent feedback lets non-verbalized computation re-enter the stack with a renewed depth budget, while preserving the standard transformer architecture, KV cache, and language-modeling objective. To train full-bandwidth transformers without losing parallel teacher forcing, we use a scheduled multi-pass objective that introduces latent feedback late in pretraining and mixes a small fraction of deeper feedback passes for stability. We train 1B-parameter full-bandwidth transformers up to 400B tokens and find that latent feedback improves validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance. With negligible per-token decoding overhead, full-bandwidth transformers match or approach standard transformers trained with roughly $1.5\times$ more tokens, and manage to produce shorter reasoning traces at equal or better accuracy.

Problem

• autoregressive transformers
• vertical feedback
• latent feedback
• decoding steps
• hidden state reuse

Innovation

• full-bandwidth transformer
• latent feedback
• autoregressive modeling
• gated linear unit
• multi-pass training