Free Pause Tokens

2026-09-03

AI Summary

研究通过引入自由暂停令牌，在不增加序列长度的情况下提高语言模型的预测能力，主要在训练阶段增加少量计算成本。

Abstract

A free pause token gives a language model extra compute to form each next-token prediction (as a pause, or thinking, token does) but carries that compute in a parallel prediction stream over a weight-shared backbone rather than as an extra token in the sequence. It improves next-token prediction by 2-3 centinats in practice on a 1B parameter model. Because the pause rides an existing position instead of adding one, it is free to use: at inference it adds no context length, no KV cache, and essentially no latency with the growth in inference flops typically irrelevant as it is not the active bottleneck on throughput. The only primary cost is in training, where additional training compute versus an optimized pretraining pipeline is reduced to as low as x1.14 while preserving most of the benefits. The result is an isoflop, isoparameter, and isotoken improvement over standard next token trained transformers.

Problem

• language model
• next-token prediction
• compute efficiency
• inference latency

Innovation

• Free Pause Token
• Weight-Shared Backbone
• Next-Token Prediction