Learning Structured Reasoning via Tractable Trajectory Control

2026-03-02

AI Summary

This work addresses the challenge that large language models struggle to generate diverse and complex reasoning trajectories under unconstrained sampling, and standard reinforcement learning (RL) fails to effectively learn such behaviors. To overcome this, the authors propose the Ctrl-R framework, which integrates a structured reasoning paradigm with a trajectory control mechanism to guide RL toward targeted exploration of specific reasoning patterns. By incorporating importance sampling weights with power-law scaling, Ctrl-R enables selective yet unbiased policy optimization over out-of-distribution trajectories. This approach maintains training stability while significantly enhancing reasoning diversity, successfully internalizing complex reasoning strategies in mathematical and vision-language tasks, and thereby achieving substantial performance gains.

Technology Category

• Machine Learning: Reinforcement Learning
• Intelligent Robots: Behavior Learning & Control
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Large language models can exhibit emergent reasoning behaviors, often manifested as recurring lexical patterns (e.g., "wait," indicating verification). However, complex reasoning trajectories remain sparse in unconstrained sampling, and standard RL often fails to guarantee the acquisition of diverse reasoning behaviors. We propose a systematic discovery and reinforcement of diverse reasoning patterns through structured reasoning, a paradigm that requires targeted exploration of specific reasoning patterns during the RL process. To this end, we propose Ctrl-R, a framework for learning structured reasoning via tractable trajectory control that actively guides the rollout process, incentivizing the exploration of diverse reasoning patterns that are critical for complex problem-solving. The resulting behavior policy enables accurate importance-sampling estimation, supporting unbiased on-policy optimization. We further introduce a power-scaling factor on the importance-sampling weights, allowing the policy to selectively learn from exploratory, out-of-distribution trajectories while maintaining stable optimization. Experiments demonstrate that Ctrl-R enables effective exploration and internalization of previously unattainable reasoning patterns, yielding consistent improvements across language and vision-language models on mathematical reasoning tasks.

Problem

• structured reasoning
• reasoning trajectories
• diverse reasoning patterns
• reinforcement learning
• trajectory control

Innovation

• structured reasoning
• tractable trajectory control
• importance sampling
• reinforcement learning
• reasoning patterns