ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning

2026-03-16

AI Summary

This work addresses the challenge that current language models struggle to effectively accumulate and reuse emergent reasoning strategies acquired during training for mathematical problem solving. To overcome this limitation, the authors propose a hierarchical reinforcement learning framework that integrates, for the first time, an evolvable skill library with hierarchical reinforcement learning. In this architecture, a high-level module maintains a dynamic skill repository through structured induction and strategy-driven retrieval, while a low-level module generates reasoning steps grounded in the retrieved skills. The two components are jointly optimized via a hierarchical reward mechanism. The approach enables cross-task strategy reuse and significantly outperforms GRPO-family algorithms and memory-augmented baselines across two base models and seven mathematical reasoning benchmarks, demonstrating particularly strong performance on out-of-distribution tasks.

Technology Category

• Machine Learning: Evolutionary Learning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The dominant paradigm for improving mathematical reasoning in language models relies on Reinforcement Learning with verifiable rewards. Yet existing methods treat each problem instance in isolation without leveraging the reusable strategies that emerge and accumulate during training. To this end, we introduce ARISE (Agent Reasoning via Intrinsic Skill Evolution), a hierarchical reinforcement learning framework, in which a shared policy operates both to manage skills at high-level and to generate responses at low-level (denoted as a Skills Manager and a Worker, respectively). The Manager maintains a tiered skill library through a dedicated skill generation rollout that performs structured summarization of successful solution traces (after execution), while employing a policy-driven selection mechanism to retrieve relevant skills to condition future rollouts (before execution). A hierarchical reward design guides the co-evolution of reasoning ability and library quality. Experiments on two base models and seven benchmarks spanning both competition mathematics and Omni-MATH show that ARISE consistently outperforms GRPO-family algorithms and memory-augmented baselines, with particularly notable gains on out-of-distribution tasks. Ablation studies confirm that each component contributes to the observed improvements and that library quality and reasoning performance improve in tandem throughout training. Code is available at \href{https://github.com/Skylanding/ARISE}{https://github.com/Skylanding/ARISE}.

Problem

• mathematical reasoning
• reusable strategies
• language models
• reinforcement learning
• skill reuse

Innovation

• Hierarchical Reinforcement Learning
• Skill Library Evolution
• Mathematical Reasoning
• Policy-Driven Skill Selection
• Structured Summarization