Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation

2026-08-10

AI Summary

This work addresses the inefficiency in reinforcement learning when group trajectory rewards are all-or-nothing, which deprives the agent of informative relative signals. To overcome this, the authors propose SKALD, a novel framework that leverages abstract skills as dense supervision signals. SKALD employs online self-distillation within a single Qwen3-Base model by constructing dual views: a “question-only” student and a “skill-card-guided” teacher, enabling skill-aware knowledge injection into shared weights without requiring privileged inputs at test time. The method further incorporates an experience-gated mechanism and an annealed exponentially tilted objective to mitigate distribution shift and uninformative rewards. Evaluated across five mathematical benchmarks, SKALD achieves average@8 improvements of +2.46, +4.85, and +12.01 on 0.6B, 1.7B, and 4B models, respectively; notably, zero-variance distillation recovers 84.7% of the gain for the 1.7B model, substantially outperforming GRPO, FLOP-matched baselines, and context-based skill exposure approaches.

Abstract

Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63.0-68.0% of groups in our experiments. We propose SKALD (Skill-Anchored Latent Distillation), an on-policy self-distillation framework that uses two context views of the same Qwen3-Base model: a question-only student and a teacher conditioned on an abstract, explicit-answer-filtered skill card. The student is trained on its own prefixes, transferring the skill-induced advantage into shared parameters without privileged input at test time. To stabilize context-induced distribution mismatch, SKALD employs an annealed exponentially tilted objective that downweights teacher-preferred tokens with very low student likelihood; as the tilt vanishes, it converges to teacher cross-entropy and recovers the forward-KL student gradient. An empirical gate activates distillation only when verified rollouts estimate a positive teacher advantage. Across five held-out mathematics benchmarks, SKALD improves overall avg@8 over GRPO by +2.46, +4.85, and +12.01 at 0.6B, 1.7B, and 4B, respectively. At 1.7B, zero-variance-only distillation recovers 84.7% of the full gain, while SKALD remains +4.06 above FLOP-matched GRPO and exceeds contextual skill exposure by +3.77. These results show that abstract skills provide dense supervision where group-relative rewards become uninformative.

Problem

• reinforcement learning
• group-relative signal
• self-distillation
• abstract skills
• reward sparsity

Innovation

• self-distillation
• abstract skills
• on-policy RL
• distribution mismatch
• privileged signals