Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling

2026-04-29

AI Summary

Existing approaches to controlling generation length operate only at the sequence level with coarse granularity, struggling to balance inference cost and performance. This work proposes LenVM, the first token-level length value modeling framework, which reframes length control as a value estimation problem. By assigning a constant negative reward to each token, LenVM constructs a bounded discounted return that serves as a monotonic proxy signal for remaining length, enabling annotation-free, dense, and scalable token-level length prediction. The method provides an unbiased supervisory signal, supporting continuously controllable and dynamically interpretable generation lengths. Experiments show that LenVM improves the length-matching score of a 7B model on LIFEBench from 30.9 to 64.8, substantially outperforming state-of-the-art closed-source models; under a 200-token budget on GSM8K, it maintains 63% accuracy (versus 6% for the baseline) and accurately predicts total output length directly from the prompt.

Abstract

Token serves as the fundamental unit of computation in modern autoregressive models, and generation length directly influences both inference cost and reasoning performance. Despite its importance, existing approaches lack fine-grained length modeling, operating primarily at the coarse-grained sequence level. We introduce the Length Value Model (LenVM), a token-level framework that models the remaining generation length. By formulating length modeling as a value estimation problem and assigning a constant negative reward to each generated token, LenVM predicts a bounded, discounted return that serves as a monotone proxy for the remaining generation horizon. This formulation yields supervision that is annotation-free, dense, unbiased, and scalable. Experiments on LLMs and VLMs demonstrate LenVM provides a highly effective signal at inference time. On the LIFEBench exact length matching task, applying LenVM to a 7B model improves the length score from 30.9 to 64.8, significantly outperforming frontier closed-source models. Furthermore, LenVM enables continuous control over the trade off between performance and efficiency. On GSM8K at a budget of 200 tokens, LenVM maintains 63% accuracy compared to 6 percent for token budget baseline. It also accurately predicts total generation length from the prompt boundary. Finally, LenVM's token-level values offer an interpretable view of generation dynamics, revealing how specific tokens shift reasoning toward shorter or longer regimes. Results demonstrate that LenVM supports a broad range of applications and token length can be effectively modeled as a token-level value signal, highlighting the potential of LenVM as a general framework for length modeling and as a length-specific value signal that could support future RL training. Code is available at https://github.com/eric-ai-lab/Length-Value-Model.

Problem

• length modeling
• token-level generation
• autoregressive models
• generation horizon
• value estimation

Innovation

• Length Value Model
• token-level length modeling
• value estimation
• annotation-free supervision
• generation control