Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

2026-08-10

AI Summary

This work addresses the challenge that existing large language model agents struggle to effectively supervise the use of external natural language skills due to reliance solely on task-level rewards in reinforcement learning. To overcome this limitation, the paper proposes a Bidirectional Contextual Self-Distillation (BCSD) framework, which introduces, for the first time, a dual-view mechanism comprising augmentation and reduction perspectives. The augmentation view injects meta-skill guidance, while the reduction view focuses on task-specific skills, jointly generating complementary token-level supervision signals to recalibrate the advantage function. This approach significantly enhances policy learning for skill-oriented agents, achieving state-of-the-art performance across model scales on the ALFWorld and WebShop benchmarks. Ablation studies further confirm that the dual-view design is crucial for improving skill utilization efficiency.

Abstract

External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks. Yet their effectiveness depends not only on skill quality, but also on whether the policy can translate the provided guidance into appropriate actions. However, methods specifically designed to improve this skill-utilization ability remain largely underexplored. In practice, skill-based agents are commonly trained with reinforcement learning objectives centered on task-level rewards, which offer limited supervision and struggle to capture subtle differences in how effectively the policy uses the provided skills. We propose BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views. Code will be released to ensure full reproducibility.

Problem

• skill utilization
• reinforcement learning
• large language model agents
• external skills
• task-level rewards

Innovation

• Bidirectional Context Self-Distillation
• Skill-Based LLM Agents
• Reinforcement Learning
• Self-Distillation
• Meta-Skill Guidance