Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key

2026-05-07

AI Summary

This work addresses the lack of controllable and scalable environments for systematically investigating how reinforcement learning (RL) enhances the long-horizon reasoning capabilities of large language models across varying task difficulties. To this end, we introduce ScaleLogic—the first synthetic reasoning benchmark that independently modulates reasoning depth and logical expressivity—and conduct a systematic evaluation across a multi-tiered hierarchy spanning propositional to first-order logic, integrating diverse RL algorithms with curriculum learning strategies. Our experiments reveal a universal power-law relationship between training compute and reasoning depth, which varies monotonically with logical expressivity. Training with higher expressivity yields up to +10.66 points in downstream performance and improved computational efficiency, while curriculum learning substantially enhances scaling efficiency.

Abstract

Reinforcement learning (RL) has been applied to improve large language model (LLM) reasoning, yet the systematic study of how training scales with task difficulty has been hampered by the lack of controlled, scalable environments. We introduce ScaleLogic, a synthetic logical reasoning framework that offers independent control over two axes of difficulty: the depth of the required proof planning (i.e., the horizon) and the expressiveness of the underlying logic. Our proposed framework supports a wide range of logics: from simple implication-only logic ("if-then") towards more expressive first-order reasoning with conjunction ("and"), disjunction ("or"), negation ("not"), and universal quantification ("for all"). Using this framework, we show that the RL training compute $T$ follows a power law with respect to reasoning depth $D$ ($T \propto D^γ$, $R^{2} > 0.99$), and that the scaling exponent $γ$ increases monotonically with logical expressiveness, from $1.04$ to $2.60$. On downstream mathematics and general reasoning benchmarks, more expressive training settings yield both larger performance gains (up to $+10.66$ points) and more compute-efficient transfer compared to less expressive settings, demonstrating that what a model is trained on, not just how much it is trained, shapes downstream transfer. We further show that the power-law relationship holds across multiple RL methods, and curriculum-based training substantially improves scaling efficiency.

Problem

• reinforcement learning
• large language models
• long-horizon reasoning
• logical expressiveness
• scaling laws

Innovation

• ScaleLogic
• reinforcement learning
• logical expressiveness
• reasoning depth
• power-law scaling