Scheduling Your LLM Reinforcement Learning with Reasoning Trees

2025-10-28

AI Summary

Existing RLVR data scheduling methods rely on path-level metrics to rank queries, neglecting their intrinsic reasoning tree structure and thus failing to accurately characterize learning difficulty. This work proposes a novel reasoning-tree-structured data scheduling paradigm for RLVR: it models LLM reinforcement learning as progressive editing of reasoning trees, introduces a computable reasoning score (r-score) to quantify query difficulty, and designs Re-Schedule—a structure-aware curriculum learning algorithm that explicitly incorporates reasoning tree topology into RLVR scheduling for the first time. The method integrates reasoning tree modeling, dynamic policy adaptation, and structured curriculum learning. Evaluated on six mathematical reasoning benchmarks, it achieves substantial average accuracy gains—up to +3.2%—demonstrating that structured scheduling meaningfully enhances both learning efficiency and generalization capability.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Using Reinforcement Learning with Verifiable Rewards (RLVR) to optimize Large Language Models (LLMs) can be conceptualized as progressively editing a query's `Reasoning Tree'. This process involves exploring nodes (tokens) and dynamically modifying the model's policy at each node. When combined with data scheduling, this process yields further gains in data efficiency and accuracy. However, existing RLVR data scheduling methods typically rely on path-based metrics to rank queries, overlooking the reasoning tree structures of these queries. In this paper, we introduce a novel metric, namely Reasoning Score (r-score), which measures the query's learning difficulty based on the structure of its reasoning tree. Based on the r-score, we propose the Reasoning Tree Schedule (Re-Schedule), a scheduling algorithm that constructs a curriculum progressing from structurally simple (high r-score) to complex (low r-score) queries. Experiments on six math-reasoning benchmarks show that Re-Schedule significantly improves average accuracy, achieving gains of up to 3.2%. These strong results validate our approach and demonstrate that a structural understanding of the reasoning tree provides a more powerful and principled foundation for RLVR data scheduling.

Problem

• Optimizing LLMs using RLVR by editing reasoning tree structures
• Existing methods ignore tree structures when scheduling query data
• Proposing structural metric and algorithm to improve learning efficiency

Innovation

• Introduces Reasoning Score metric for query difficulty
• Proposes Reasoning Tree Schedule algorithm for curriculum
• Uses tree structure to improve RLVR data scheduling