Dynamic Rollout Editing for Reducing Overthinking in RL-Trained Reasoning Models

2026-06-16

AI Summary

This work addresses the issue of redundant reasoning—termed “overthinking”—in reinforcement learning (RL)-trained reasoning models, where models continue generating unnecessary steps even after arriving at the correct answer. The study formulates this phenomenon as a credit assignment bias during training and introduces a dynamic trajectory editing mechanism within the GRPO RL framework. By integrating trajectory sampling, prefix validation, and dynamic editing, the method preserves informative reasoning prefixes while attenuating positive reward signals assigned to redundant segments, thereby avoiding undue penalization of essential reasoning steps. Experimental results demonstrate that the proposed approach effectively suppresses overthinking across diverse complex tasks, simultaneously enhancing both reasoning efficiency and overall model performance.

Abstract

Long-form chain-of-thought reasoning can improve LLM performance on complex tasks, but models often continue generating unnecessary reasoning after a correct answer has emerged. We refer to this behavior as overthinking. We study this phenomenon from the perspective of GRPO-style reinforcement learning (RL) post-training, framing it as a training-time credit-assignment problem rather than merely a decoding-time stopping problem. In rollouts sampled at the onset of GRPO training, we observe that successful trajectories can exhibit a slightly higher degree of overthinking than unsuccessful trajectories for the same prompts. This early imbalance provides a starting point for an undesirable feedback loop: because GRPO assigns sequence-level credit, it cannot distinguish the solution-reaching prefix from the unnecessary continuation that lengthens a successful trajectory. Both receive positive update signal, allowing the initial imbalance to grow into more severe overthinking during training. To address this issue, we introduce Dynamic Rollout Editing (DRE), a training-time intervention for successful trajectories that continue thinking after answer emergence. DRE preserves the accepted verified prefix, edits the remaining thinking, and prefers the edited trajectory within the same RL group, weakening the preference signal for unnecessary thinking without penalizing the reasoning needed to reach the answer. Experiments across diverse tasks show the effectiveness of DRE.

Problem

• overthinking
• reinforcement learning
• credit assignment
• chain-of-thought reasoning
• RL-trained reasoning models

Innovation

• Dynamic Rollout Editing
• overthinking
• reinforcement learning
• credit assignment
• reasoning models