Selected work

Representative Papers

Addressing Over-Refusal in LLMs with Competing Rewards

Jun 30, 2026

This work addresses the issue of over-refusal in large language models during safety alignment training, where harmless user requests are erroneously rejected. To mitigate this, the authors propose SEAR, a novel framework that introduces dual roles—“reasoning player” and “response player”—within a single chain-of-thought process. The reasoning player proactively explores potentially harmful reasoning paths to generate adversarial signals, while the response player produces safe and compliant outputs. Leveraging reinforcement learning with dense process-level rewards, SEAR jointly optimizes both roles within a unified architecture, enabling precise discrimination between harmful and benign inputs. Experimental results demonstrate that SEAR significantly alleviates over-refusal and enhances robustness against attacks that directly manipulate the model’s reasoning process.

0 citationsRead paper

Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning

Jun 05, 2025

Large reasoning models (LRMs) frequently generate redundant tokens on simple problems, wasting computational resources; existing length-control methods—such as supervised fine-tuning or fixed-penalty reinforcement learning—fail to adapt to varying problem difficulty. This work proposes Adaptive Length Penalty (ALP), a mechanism that dynamically adjusts the per-token generation cost based on the online solving rate estimated for each prompt, enabling difficulty-aware length control. ALP integrates differentiable reinforcement learning, rolling online evaluation, and post-training fine-tuning (using DeepScaleR-1.5B). Experiments demonstrate that ALP reduces average token consumption by 50% without statistically significant accuracy degradation overall—and even improves accuracy on the most challenging problems. To our knowledge, this is the first approach achieving fine-grained, data-agnostic, difficulty-adaptive generation length control specifically designed for LRMs.

0 citationsRead paper

Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs

Mar 03, 2025

Why do some language models exhibit sustained self-improvement under reinforcement learning (RL), while others rapidly plateau? This work identifies the initial reasoning behavior pattern—not answer correctness—as the critical determinant. Specifically, four cognitive behaviors—verification, backtracking, subgoal decomposition, and backward chaining—prove decisive for both test-time reasoning and RL efficacy. We propose behavior-oriented prompt augmentation and a filtering-based continued pretraining methodology using OpenWebMath, enabling targeted behavioral supervision. Additionally, we construct a controllable behavioral dataset to isolate and modulate these reasoning patterns. Experiments show that Llama-3.2-3B, optimized via our framework, achieves self-improvement performance on the Countdown task comparable to—and in some cases exceeding—that of Qwen-2.5-3B. Crucially, even incorrect answers containing valid reasoning structures effectively drive RL learning, confirming the primacy of reasoning process over outcome in model self-refinement.

0 citationsRead paper

Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models

Feb 24, 2025

Existing mathematical datasets face a trade-off between quality and scale: high-quality human-authored problems are scarce, while machine-generated problems suffer from uncontrolled quality, hindering reinforcement learning (RL) training for language models. To address this, we propose MathRL—a large-scale, high-quality mathematics dataset specifically designed for RL. It comprises 253,000 problems, all manually verified end-to-end, spanning diverse domains and a broad difficulty spectrum. We introduce a novel three-tier quality filtering framework, integrating a rule-based reformulation algorithm—yielding 47,000 new open-ended problems—and a multi-source fusion cleaning pipeline. MathRL exceeds GSM8K and MATH in scale by over an order of magnitude, while ensuring answer verifiability, problem openness, and closed-form solvability. Experiments demonstrate substantial improvements in RL training stability and cross-task generalization performance.

0 citationsRead paper

Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Though

Jan 08, 2025

This work addresses the lack of human-like systematic reasoning—akin to System 2 thinking—in large language models (LLMs). To this end, we propose Meta-CoT, the first framework that elevates chain-of-thought (CoT) reasoning from answer generation to explicit modeling of the reasoning process itself. Meta-CoT explicitly learns and generates interpretable, verifiable, and optimizable reasoning paths—not merely final answers. Methodologically, it introduces an end-to-end training pipeline integrating process-supervised fine-tuning, synthetic reasoning trajectory generation, linearized search instruction tuning, and reinforcement learning. Evaluated on multiple complex reasoning benchmarks, Meta-CoT exhibits emergent implicit search behavior, yielding substantial improvements in both generalization and interpretability. This work establishes a novel paradigm and practical pathway toward controllable, deep, and traceable System 2 reasoning in LLMs.

0 citationsRead paper

Recent publications

Latest Papers

Addressing Over-Refusal in LLMs with Competing Rewards

Jun 30, 2026

This work addresses the issue of over-refusal in large language models during safety alignment training, where harmless user requests are erroneously rejected. To mitigate this, the authors propose SEAR, a novel framework that introduces dual roles—“reasoning player” and “response player”—within a single chain-of-thought process. The reasoning player proactively explores potentially harmful reasoning paths to generate adversarial signals, while the response player produces safe and compliant outputs. Leveraging reinforcement learning with dense process-level rewards, SEAR jointly optimizes both roles within a unified architecture, enabling precise discrimination between harmful and benign inputs. Experimental results demonstrate that SEAR significantly alleviates over-refusal and enhances robustness against attacks that directly manipulate the model’s reasoning process.

0 citationsRead paper

Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning

Jun 05, 2025

Large reasoning models (LRMs) frequently generate redundant tokens on simple problems, wasting computational resources; existing length-control methods—such as supervised fine-tuning or fixed-penalty reinforcement learning—fail to adapt to varying problem difficulty. This work proposes Adaptive Length Penalty (ALP), a mechanism that dynamically adjusts the per-token generation cost based on the online solving rate estimated for each prompt, enabling difficulty-aware length control. ALP integrates differentiable reinforcement learning, rolling online evaluation, and post-training fine-tuning (using DeepScaleR-1.5B). Experiments demonstrate that ALP reduces average token consumption by 50% without statistically significant accuracy degradation overall—and even improves accuracy on the most challenging problems. To our knowledge, this is the first approach achieving fine-grained, data-agnostic, difficulty-adaptive generation length control specifically designed for LRMs.

0 citationsRead paper

Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs

Mar 03, 2025

Why do some language models exhibit sustained self-improvement under reinforcement learning (RL), while others rapidly plateau? This work identifies the initial reasoning behavior pattern—not answer correctness—as the critical determinant. Specifically, four cognitive behaviors—verification, backtracking, subgoal decomposition, and backward chaining—prove decisive for both test-time reasoning and RL efficacy. We propose behavior-oriented prompt augmentation and a filtering-based continued pretraining methodology using OpenWebMath, enabling targeted behavioral supervision. Additionally, we construct a controllable behavioral dataset to isolate and modulate these reasoning patterns. Experiments show that Llama-3.2-3B, optimized via our framework, achieves self-improvement performance on the Countdown task comparable to—and in some cases exceeding—that of Qwen-2.5-3B. Crucially, even incorrect answers containing valid reasoning structures effectively drive RL learning, confirming the primacy of reasoning process over outcome in model self-refinement.

0 citationsRead paper

Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models

Feb 24, 2025

Existing mathematical datasets face a trade-off between quality and scale: high-quality human-authored problems are scarce, while machine-generated problems suffer from uncontrolled quality, hindering reinforcement learning (RL) training for language models. To address this, we propose MathRL—a large-scale, high-quality mathematics dataset specifically designed for RL. It comprises 253,000 problems, all manually verified end-to-end, spanning diverse domains and a broad difficulty spectrum. We introduce a novel three-tier quality filtering framework, integrating a rule-based reformulation algorithm—yielding 47,000 new open-ended problems—and a multi-source fusion cleaning pipeline. MathRL exceeds GSM8K and MATH in scale by over an order of magnitude, while ensuring answer verifiability, problem openness, and closed-form solvability. Experiments demonstrate substantial improvements in RL training stability and cross-task generalization performance.

0 citationsRead paper

Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Though

Jan 08, 2025

This work addresses the lack of human-like systematic reasoning—akin to System 2 thinking—in large language models (LLMs). To this end, we propose Meta-CoT, the first framework that elevates chain-of-thought (CoT) reasoning from answer generation to explicit modeling of the reasoning process itself. Meta-CoT explicitly learns and generates interpretable, verifiable, and optimizable reasoning paths—not merely final answers. Methodologically, it introduces an end-to-end training pipeline integrating process-supervised fine-tuning, synthetic reasoning trajectory generation, linearized search instruction tuning, and reinforcement learning. Evaluated on multiple complex reasoning benchmarks, Meta-CoT exhibits emergent implicit search behavior, yielding substantial improvements in both generalization and interpretability. This work establishes a novel paradigm and practical pathway toward controllable, deep, and traceable System 2 reasoning in LLMs.

0 citationsRead paper