Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning

2026-05-18

AI Summary

Existing reinforcement learning approaches for code reasoning rely on sparse or coarse-grained rewards, often neglecting the intrinsic logical consistency of the reasoning process and thereby suffering from reward sparsity and deceptive optimization. This work proposes CodeThinker, a novel framework that introduces consistency-driven reinforcement learning to code reasoning for the first time. By integrating stepwise reasoning-aware training, dynamic beam sampling, and a consistency-based reward mechanism, CodeThinker effectively models the logical coherence of reasoning chains. Without any additional training, the method achieves state-of-the-art performance across three major benchmarks, improving the accuracy of Qwen2.5-Coder-7B-Instruct by 4.3%. Furthermore, it yields average gains of 5.33 and 3.11 percentage points on downstream tasks in mathematical and multilingual code reasoning, respectively.

Abstract

Code reasoning refers to the task of predicting the output of a program given its source code and specific inputs. It can measure the reasoning capability of large language models (LLMs) and also benefit downstream tasks such as code generation and mathematical reasoning. Existing work has verified the effectiveness of reinforcement learning on the task. However, these methods design rewards solely based on final outputs or coarse-grained signals, and neglect the inherent consistency of the stepwise reasoning process in the task. Therefore, these methods often result in sparse reward or reward hacking, which limits the full play of enhanced learning capabilities. To alleviate these issues, we propose CodeThinker, a consistency-driven reinforcement learning framework for code reasoning. Specifically, CodeThinker has three key components: (1) a stepwise reasoning-aware model training module, which utilizes a consistency tracing paradigm as a template to synthesize training data that captures the stepwise reasoning process; (2) a dynamic beam sampling strategy, which aims to improve the quality of sampled outputs under a fixed sampling budget; and (3) a consistency reward mechanism that can effectively alleviate reward hacking. Experiments on three popular benchmarks show that CodeThinker achieves state-of-the-art performance across multiple LLMs. For instance, it outperforms the strongest baseline by 4.3% in accuracy when deployed on Qwen2.5-Coder-7B-Instruct. We also validate the effectiveness of CodeThinker on downstream tasks. Results show that, without additional training, CodeThinker obtains average accuracy gains of 5.33 and 3.11 percentage points on mathematical reasoning and code reasoning tasks covering 17 programming languages, respectively.

Problem

• code reasoning
• reinforcement learning
• reward sparsity
• reward hacking
• stepwise reasoning consistency

Innovation

• consistency-based reinforcement learning
• code reasoning
• stepwise reasoning
• reward hacking mitigation
• dynamic beam sampling