Thinking-Based Non-Thinking: Solving the Reward Hacking Problem in Training Hybrid Reasoning Models via Reinforcement Learning

2026-01-08

arXiv.org

AI Summary

This work addresses the susceptibility of hybrid reasoning models in reinforcement learning to “reward hacking,” wherein models misattribute reasoning behavior, leading to suboptimal decisions. The authors propose a training method that requires no supervised fine-tuning and instead dynamically analyzes solution-relevant information within reasoning trajectories to adaptively cap the maximum generation length for non-reasoning responses per query. This approach effectively mitigates reward hacking while maintaining computational efficiency. Evaluated across five mathematical benchmarks, the method significantly outperforms DeepSeek-R1-Distill-Qwen-1.5B/7B and DeepScaleR-1.5B, achieving higher accuracy while reducing token consumption by approximately 50% and constraining the probability of reward hacking to below 10%.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Metareasoning and Metaheuristics
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large reasoning models (LRMs) have attracted much attention due to their exceptional performance. However, their performance mainly stems from thinking, a long Chain of Thought (CoT), which significantly increase computational overhead. To address this overthinking problem, existing work focuses on using reinforcement learning (RL) to train hybrid reasoning models that automatically decide whether to engage in thinking or not based on the complexity of the query. Unfortunately, using RL will suffer the the reward hacking problem, e.g., the model engages in thinking but is judged as not doing so, resulting in incorrect rewards. To mitigate this problem, existing works either employ supervised fine-tuning (SFT), which incurs high computational costs, or enforce uniform token limits on non-thinking responses, which yields limited mitigation of the problem. In this paper, we propose Thinking-Based Non-Thinking (TNT). It does not employ SFT, and sets different maximum token usage for responses not using thinking across various queries by leveraging information from the solution component of the responses using thinking. Experiments on five mathematical benchmarks demonstrate that TNT reduces token usage by around 50% compared to DeepSeek-R1-Distill-Qwen-1.5B/7B and DeepScaleR-1.5B, while significantly improving accuracy. In fact, TNT achieves the optimal trade-off between accuracy and efficiency among all tested methods. Additionally, the probability of reward hacking problem in TNT's responses, which are classified as not using thinking, remains below 10% across all tested datasets.

Problem

• reward hacking
• hybrid reasoning models
• reinforcement learning
• Chain of Thought
• computational overhead

Innovation

• reward hacking
• hybrid reasoning models
• Chain of Thought
• reinforcement learning
• token efficiency