Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning

2026-06-17

AI Summary

This work addresses the overreliance on intricate reward engineering in existing reinforcement learning (RL) approaches for enhancing long-context reasoning in large language models, which often neglects the role of data diversity. The authors propose a concise, data-driven framework that combines a carefully curated data recipe—comprising eight high-quality datasets (~14K samples) spanning retrieval, multi-evidence integration, and reasoning tasks—with a lightweight, outcome-oriented GRPO algorithm. Evaluated on the Qwen3 model series, this approach achieves average gains of +7.2, +3.2, and +6.4 points across seven long-context benchmarks, substantially outperforming current RL methods. It also improves performance by +4.8 and +7.0 points on the GAIA and BrowseComp agent tasks, respectively, providing the first empirical validation that data diversity and task complementarity are pivotal for advancing long-context reasoning capabilities.

Abstract

Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.

Problem

• long-context reasoning
• reinforcement learning
• training data
• reward engineering
• large language models

Innovation

• data-centric reinforcement learning
• long-context reasoning
• GRPO
• multi-evidence synthesis
• retrieval-augmented tasks