OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation

2026-06-16

AI Summary

While existing memory-augmented agents can store experiences, they struggle to effectively filter, apply, distill, and maintain reusable knowledge. This work proposes a slow-fast co-evolution framework: in the fast loop, test-time adaptation is achieved through intra-policy self-distillation and interaction across a four-level memory hierarchy; in the slow loop, memory attribution calibrated by outcome feedback and privileged post-hoc knowledge distillation transfer four core capabilities—experience filtering, action selection, knowledge encoding, and memory maintenance—to the deployed policy. This mechanism unifies these four dimensions for the first time, advancing agents from mere memory augmentation toward continual evolution. Experiments show that the model outperforms ReasoningBank by 11.5% and Skill0 by 5.8% across multiple benchmarks, with OPD-Evolver-9B matching the performance of much larger models such as Qwen3.5-397B-A17B.

Abstract

Memory has become a standard substrate for self-evolving agents, yet retaining experience is not the same as learning how to evolve through it. Existing memory agents can store trajectories, retrieve reflections, or accumulate skills, but often lack the holistic competence to select useful experience, act on it, write reusable knowledge, and maintain a growing repository. We introduce OPD-Evolver, a slow-fast co-evolution framework that cultivates such an agent evolver through on-policy self-distillation. In the fast loop, OPD-Evolver interacts with a four-level memory hierarchy to read, use, write, and maintain experience for rapid test-time evolution. In the slow loop, outcome-calibrated memory attribution and privileged hindsight distill these four abilities into the deployable policy. Across multi-domain benchmarks, OPD-Evolver surpasses memory systems such as ReasoningBank by up to 11.5%, and training-based methods such as Skill0 by ~5.8%. Further analysis shows that OPD-Evolver internalizes high-value experience and memory management, enabling OPD-Evolver-9B to challenge giant counterparts such as Qwen3.5-397B-A17B and Step-3.5-Flash, pointing beyond memory-augmented agents toward genuinely qualified agent evolvers.

Problem

• agent evolution
• memory management
• experience utilization
• knowledge reuse
• holistic competence

Innovation

• on-policy distillation
• agent evolver
• memory hierarchy
• self-evolution
• slow-fast co-evolution