Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models

2025-10-04

AI Summary

To address the pervasive “overthinking” problem in Large Reasoning Models (LRMs)—characterized by unnecessarily lengthy reasoning chains that impair efficiency and exacerbate speculative pruning—this paper proposes a step-level pruning reinforcement learning framework. Unlike conventional token-count-based sparsity penalties, our approach operates at the reasoning-step granularity: we design a step-aware reward function and a dynamic stopping mechanism to explicitly decouple correctness from step redundancy, and introduce an output-length-triggered update blocking technique to mitigate policy collapse. Evaluated on four major reasoning benchmarks, our method maintains or even improves accuracy while substantially compressing reasoning length—e.g., reducing token count by 69.7% on AIME24—achieving, for the first time, step-level controllable, robust, and efficient reasoning optimization.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks but often suffer from excessive verbosity, known as "overthinking." Existing solutions via reinforcement learning (RL) typically penalize generated tokens to promote conciseness. However, these methods encounter two challenges: responses with fewer tokens do not always correspond to fewer reasoning steps, and models may develop hacking behavior in later stages of training by discarding reasoning steps to minimize token usage. In this work, we introduce extbf{Step Pruner (SP)}, an RL framework that steers LRMs toward more efficient reasoning by favoring compact reasoning steps. Our step-aware reward function prioritizes correctness while imposing penalties for redundant steps, and withholds rewards for incorrect responses to prevent the reinforcement of erroneous reasoning. Moreover, we propose a dynamic stopping mechanism: when the length of any output step exceeds the upper limit, we halt updates to prevent hacking behavior caused by merging steps. Extensive experiments across four reasoning benchmarks demonstrate that SP achieves state-of-the-art accuracy while significantly reducing response length. For instance, on AIME24, SP reduces token usage by extbf{69.7%}.

Problem

• Reducing excessive verbosity and overthinking in Large Reasoning Models
• Preventing token-based hacking behavior that sacrifices reasoning quality
• Optimizing reasoning efficiency while maintaining high accuracy in responses

Innovation

• Step Pruner RL framework promotes compact reasoning steps
• Step-aware reward penalizes redundancy and withholds for errors
• Dynamic stopping prevents step merging and hacking behavior