Implicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Training

2026-05-08

AI Summary

This work addresses the challenge in reinforcement learning–based reasoning where models often generate excessively long chains of thought due to “overthinking,” while existing compression techniques risk “underthinking” and degrade accuracy. The authors introduce a novel approach that leverages the sign of the dynamic correlation between response length and accuracy to implicitly distinguish overthinking from underthinking states. They propose an implicit compression regularization method that constructs a virtual, shorter target distribution based on the shortest correct responses already present in the policy’s output, thereby guiding the model to produce concise yet accurate reasoning trajectories. Without requiring external supervision or intervention, this method significantly reduces response length while maintaining or even improving accuracy across three backbone models and multiple math- and knowledge-intensive benchmarks, achieving a superior Pareto frontier between accuracy and response length.

Abstract

Reinforcement learning with verifiable rewards improves LLM reasoning but often induces overthinking, where models generate unnecessarily long reasoning traces. Existing methods mainly rely on length penalties or early-exit strategies; however, the former may degrade accuracy and induce underthinking, whereas the latter assumes that substantial portions of reasoning traces can be safely truncated. To obtain a compression signal without these limitations, we revisit the training dynamics of existing compression methods. We observe that the length--accuracy correlation is initially negative but continually increases during compression, indicating that shorter responses are initially more likely to be correct but gradually lose this property as the policy moves toward underthinking. Based on this observation, we formalize overthinking: a negative correlation indicates an overthinking regime, while a positive one indicates underthinking. When overthinking, the shortest correct responses are shorter than the group-average response length in expectation, making them natural compression targets already present in on-policy rollouts. We therefore propose \emph{Implicit Compression Regularization} (ICR), an on-policy regularization method whose compression signal comes from a virtual shorter distribution induced by the shortest correct responses in rollout groups, guiding the policy toward concise yet correct trajectories. Training dynamics show that ICR maintains a better length--accuracy correlation during compression, indicating that short responses remain better aligned with correctness instead of drifting toward underthinking. Experiments on three reasoning backbones and multiple mathematical and knowledge-intensive benchmarks show that ICR consistently shortens responses while preserving or improving accuracy, achieving a stronger accuracy--length Pareto frontier.

Problem

• overthinking
• reasoning compression
• length-accuracy trade-off
• reinforcement learning
• LLM reasoning

Innovation

• Implicit Compression Regularization
• overthinking
• length-accuracy correlation
• on-policy regularization
• reasoning compression