Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning

2026-02-02

AI Summary

This work addresses the inefficiency–accuracy trade-off in verifiable reward-based reinforcement learning (RLVR), where multi-step reasoning often yields overly long trajectories and static length penalties compromise correctness. The authors propose Dynamic Decoupled Conditional Advantage (DDCA), a novel method that decouples efficiency optimization from correctness by computing length advantage only over correctly answered subsets and dynamically adjusting penalty strength based on problem difficulty. DDCA resolves two structural issues—length baseline dilution and mismatch between difficulty and penalty—through conditional advantage estimation, difficulty-aware penalization, and an adaptive scaling strategy based on population pass rates. Evaluated on benchmarks including GSM8K, MATH500, AMC23, and AIME25, the approach significantly improves the efficiency–accuracy balance: reducing generated tokens by approximately 60% on easy problems and over 20% on hard ones, while maintaining or even enhancing accuracy.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) can elicit strong multi-step reasoning, yet it often encourages overly verbose traces. Moreover, naive length penalties in group-relative optimization can severely hurt accuracy. We attribute this failure to two structural issues: (i) Dilution of Length Baseline, where incorrect responses (with zero length reward) depress the group baseline and over-penalize correct solutions; and (ii) Difficulty-Penalty Mismatch, where a static penalty cannot adapt to problem difficulty, suppressing necessary reasoning on hard instances while leaving redundancy on easy ones. We propose Dynamic Decoupled Conditional Advantage (DDCA) to decouple efficiency optimization from correctness. DDCA computes length advantages conditionally within the correct-response cluster to eliminate baseline dilution, and dynamically scales the penalty strength using the group pass rate as a proxy for difficulty. Experiments on GSM8K, MATH500, AMC23, and AIME25 show that DDCA consistently improves the efficiency--accuracy trade-off relative to adaptive baselines, reducing generated tokens by approximately 60% on simpler tasks (e.g., GSM8K) versus over 20% on harder benchmarks (e.g., AIME25), thereby maintaining or improving accuracy. Code is available at https://github.com/alphadl/DDCA.

Problem

• Reinforcement Learning with Verifiable Rewards
• length penalty
• reasoning efficiency
• baseline dilution
• difficulty adaptation

Innovation

• Dynamic Decoupled Conditional Advantage
• Efficient Reasoning
• Reinforcement Learning with Verifiable Rewards
• Length Penalty
• Difficulty-Adaptive Optimization