Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning

2026-01-31

AI Summary

Large language models often struggle with complex reasoning tasks under reinforcement learning with verifiable rewards (RLVR) due to inefficient exploration. To address this, this work proposes A²D, the first adaptive problem decomposition framework that operates without teacher models or distillation. A²D introduces a trainable, plug-and-play decomposer that breaks down the original problem into verifiable subproblems, which then guide the RLVR training of the reasoning model, thereby enhancing both exploration and exploitation. The method is compatible with various RLVR algorithms and consistently outperforms existing approaches across multiple benchmarks, demonstrating its effectiveness, generality, and capacity to substantively improve the reasoning behavior of language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown great potential to enhance the reasoning ability of large language models (LLMs). However, due to the limited amount of information provided during the RLVR process, the model can only engage in largely blind exploration, which often results in failure on challenging problems. To provide additional information for the RLVR process without relying on a teacher model, we propose A$^2$D, an Adaptive Ability Decomposing method for enhancing the effectiveness of RLVR. Specifically, we first train a decomposer via RLVR without distillation, enabling it to decompose complex questions into a set of simpler sub-questions. Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and then train the reasoner under RLVR with sub-question guidance. To better understand A$^2$D, we first compare its performance with competitive baselines, showing its effectiveness. Next, we observe that our method functions as a plug-and-play module that can be applied to different RLVR algorithms. Furthermore, we conduct an analysis of the decomposer, revealing how the RLVR process affects its performance and behavior, and which type of guidance is better suited for enhancing the reasoner's exploration and exploitation abilities.

Problem

• Reinforcement Learning
• Large Language Models
• Reasoning Ability
• Reward Sparsity
• Exploration Efficiency

Innovation

• Adaptive Ability Decomposing
• Reinforcement Learning with Verifiable Rewards
• Reasoning Decomposition
• Plug-and-Play Module
• Sub-question Guidance