Agentic Policy Optimization via Instruction-Policy Co-Evolution

2025-12-01

AI Summary

Existing Reinforcement Learning with Verifiable Rewards (RLVR) approaches rely on static human-written instructions, which fail to adapt to policy evolution and environmental exploration, resulting in suboptimal instruction guidance. Method: We propose INSPO, the first framework to embed instruction optimization directly into the RL closed loop. It maintains an evolvable instruction population and introduces an online reflection mechanism grounded in experience replay, enabling co-evolution of instructions and policies. The method integrates large language model–driven RL, verifiable rewards, instruction sampling/pruning, and dynamic instruction generation. Contribution/Results: Evaluated on multi-step retrieval and reasoning tasks, INSPO significantly outperforms static-instruction baselines, discovering superior reasoning paths and tool-call sequences. It achieves substantial performance gains with only marginal computational overhead.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capability of large language models (LLMs), enabling autonomous agents that can conduct effective multi-turn and tool-integrated reasoning. While instructions serve as the primary protocol for defining agents, RLVR typically relies on static and manually designed instructions. However, those instructions may be suboptimal for the base model, and the optimal instruction may change as the agent's policy improves and explores the interaction with the environment. To bridge the gap, we introduce INSPO, a novel Instruction-Policy co-evolution framework that integrates instruction optimization as a dynamic component of the reinforcement learning (RL) loop. INSPO maintains a dynamic population of instruction candidates that are sampled with questions, where reward signals in RL loops are automatically attributed to each instruction, and low performers are periodically pruned. New instructions are generated and verified through an on-policy reflection mechanism, where an LLM-based optimizer analyzes past experience from a replay buffer and evolves more effective strategies given the current policy. We conduct extensive experiments on multi-turn retrieval and reasoning tasks, demonstrating that INSPO substantially outperforms strong baselines relying on static instructions. INSPO discovers innovative instructions that guide the agent toward more strategic reasoning paths, achieving substantial performance gains with only a marginal increase in computational overhead.

Problem

• Optimizes agent instructions dynamically during RL training
• Addresses suboptimal static instructions in RLVR for LLMs
• Co-evolves instructions and policies to enhance reasoning performance

Innovation

• Co-evolves instructions and policies dynamically
• Uses on-policy reflection to generate new instructions
• Automatically prunes low-performing instructions via rewards