UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation

2026-06-28

AI Summary

Existing agents are often misled when leveraging skill memories due to the inconsistent utility of skills across different states, rendering methods that rely on fixed skill prompts as teacher signals ineffective. This work proposes a credit-aware on-policy bidirectional self-distillation mechanism that treats skill-augmented and skill-free prompts as two contextual views of the same model. By comparing return-to-go values under identical tasks and anchored states, the view yielding higher returns serves as a local teacher to dynamically guide skill usage, memory updates, and self-training. Abandoning the assumption of a fixed teacher, this approach enables utility-aware skill retrieval, dynamic evaluation, and reflective evolution. It substantially outperforms skill-free reinforcement learning, skill-memory baselines, and existing self-distillation methods on ALFWorld, WebShop, and Search-QA, achieving gains of up to 23.5 and 18.0 percentage points.

Abstract

Skill memories can improve agentic reinforcement learning by reusing past experience as textual guidance, but retrieved skills are not oracular: they may help in one state while misleading the same policy in another. This makes the common privileged-teacher assumption fragile, namely that a skill-conditioned prompt can be treated as a fixed teacher for the no-skill prompt. We introduce UCOB, a framework for learning to utilize and evolve agentic skills via credit-aware on-policy bidirectional self-distillation. UCOB treats skill-conditioned and no-skill prompts as two on-policy context views of the same model, compares their return-to-go within the same task and anchor state, and uses the higher-return view as the local teacher. This local credit signal internalizes useful skill-conditioned behavior, corrects misleading skill usage, and guides task/state skill memory updates, utility-aware retrieval, and reflection self-training. Experiments on agentic tasks, including ALFWorld, WebShop, and Search-QA, show that UCOB outperforms skill-free RL, skill-memory baselines, and self-distillation methods across model scales, with up to 23.5 and 18.0 point gains over SOTA baselines on ALFWorld and WebShop. Ablations and analyses further validate its core mechanisms and efficiency.

Problem

• agentic reinforcement learning
• skill memories
• credit assignment
• self-distillation
• skill utilization

Innovation

• credit-aware self-distillation
• agentic reinforcement learning
• skill memory
• on-policy bidirectional distillation
• utility-aware retrieval