ETR: Entropy Trend Reward for Efficient Chain-of-Thought Reasoning

2026-04-06

AI Summary

This work addresses the inefficiency of lengthy chain-of-thought (CoT) reasoning generated by large language models, a problem exacerbated by existing compression methods that overlook the critical role of uncertainty dynamics in reasoning efficiency. The study reveals, for the first time, that reasoning efficiency is governed by the trend of uncertainty reduction rather than absolute entropy values. To exploit this insight, the authors propose Entropy Trend Reward (ETR)—a trajectory-aware optimization objective that encourages a steady global decline in uncertainty while permitting local exploration. Integrated within the GRPO framework, ETR dynamically modulates the generation process, reducing the reasoning length of DeepSeek-R1-Distill-7B by 67% across four benchmarks while simultaneously improving accuracy by 9.9%, substantially outperforming current state-of-the-art approaches.

Abstract

Chain-of-thought (CoT) reasoning improves large language model performance on complex tasks, but often produces excessively long and inefficient reasoning traces. Existing methods shorten CoTs using length penalties or global entropy reduction, implicitly assuming that low uncertainty is desirable throughout reasoning. We show instead that reasoning efficiency is governed by the trajectory of uncertainty. CoTs with dominant downward entropy trends are substantially shorter. Motivated by this insight, we propose Entropy Trend Reward (ETR), a trajectory-aware objective that encourages progressive uncertainty reduction while allowing limited local exploration. We integrate ETR into Group Relative Policy Optimization (GRPO) and evaluate it across multiple reasoning models and challenging benchmarks. ETR consistently achieves a superior accuracy-efficiency tradeoff, improving DeepSeek-R1-Distill-7B by 9.9% in accuracy while reducing CoT length by 67% across four benchmarks. Code is available at https://github.com/Xuan1030/ETR

Problem

• Chain-of-Thought
• reasoning efficiency
• large language models
• entropy
• uncertainty

Innovation

• Entropy Trend Reward
• Chain-of-Thought Reasoning
• Uncertainty Trajectory
• Reasoning Efficiency
• Policy Optimization