Experience Augmented Policy Optimization for LLM Reasoning

2026-06-29

AI Summary

This work addresses key challenges in applying reinforcement learning to enhance the reasoning capabilities of large language models, including high sampling costs, inefficient experience utilization, and policy drift. To overcome these limitations, the authors propose EAPO, a novel approach that introduces action-level experience priors and dynamically injects relevant historical experiences at critical decision points. By integrating an adapted importance sampling mechanism, EAPO enables policy-adaptive experience reuse, effectively mitigating experience mismatch and policy drift. Evaluated on five benchmark datasets using Qwen-2.5-Math-7B and Qwen-3-8B, EAPO consistently outperforms the state-of-the-art RLVR method, demonstrating significant and sustained improvements in reasoning performance.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR methods typically rely on on-policy optimization from scratch, resulting in high sampling costs and inefficient utilization of accumulated experience. As model capabilities and policy behaviors evolve during training, recent attempts to reuse experience via fixed reasoning trajectories further suffer from policy mismatch. Motivated by these limitations, we argue that experience in RLVR should not be reused as fixed reasoning trajectories, but instead expressed in a policy-adaptive manner. In this work, we propose Experience-Augmented Policy Optimization (EAPO), which leverages a prior RL-optimized policy as an action-level experience prior and selectively injects experience at critical decision points during rollout. To ensure stable and unbiased learning from experience-augmented rollouts, EAPO further incorporates an adapted importance sampling scheme. Experiments on using Qwen-2.5-math 7b and Qwen-3-8B on five different benchmarks demonstrate that EAPO consistently improves reasoning performance over state-of-the-art RLVR methods.

Problem

• Reinforcement Learning with Verifiable Rewards
• experience reuse
• policy mismatch
• sampling efficiency
• large language models

Innovation

• Experience-Augmented Policy Optimization
• Reinforcement Learning with Verifiable Rewards
• policy-adaptive experience reuse
• importance sampling
• LLM reasoning