APEx: Distillation of Agent Procedural Experience for Adaptive Deep Research Question Answering

2026-09-02

AI Summary

APEx通过组织交互历史并使用Executor、Distiller和Planner的闭环架构，解决深度研究代理持续改进的问题，实现技能蒸馏与策略适应。

Abstract

Deep research agents augment large language models with external tools to answer complex, long-horizon questions through multi-turn reasoning. Learning from prior experience is crucial for continual improvement, yet existing methods either retrieve verbose task-specific traces that burden decision-making, or distill procedural skills that remain decoupled from downstream policy adaptation. We propose APEx, a hierarchical experience utilization framework that organizes interaction history into instance-level trajectory memories and category-level procedural skills, and couples them through a closed-loop architecture of Executor, Distiller, and Planner. The three modules are optimized via a three-stage alternating GRPO training paradigm, enabling reward-guided skill distillation rather than fixed-prompt generation. At test time, distilled skills serve as procedural priors for online Planner adaptation through skill-guided test-time reinforcement learning, allowing ground-truth-free self-improvement with skill-alignment regularization to prevent policy drift. Experiments on 7 benchmarks demonstrate that APEx achieves state-of-the-art performance, surpassing GPT-5.4 by 14.7 points and the strongest memory-augmented baseline by 3.0 points.

Problem

• Deep research agents
• Procedural experience
• Policy adaptation
• Experience utilization
• Skill distillation

Innovation

• Hierarchical Experience Utilization
• Reward-Guided Skill Distillation
• Skill-Aligned Regularization