GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering

2026-05-28

AI Summary

This work addresses the challenge in reinforcement learning–based knowledge base question answering, where sparse rewards based solely on final answers provide insufficient supervision for intermediate reasoning steps. To overcome this limitation, the authors propose the GAPD framework, which innovatively leverages token-level dense supervision derived from the action sequence of a gold logical form. Central to this approach is the MID-ANCHOR MATCHING mechanism, which aligns exploration trajectories with gold actions at intermediate entities and integrates a gradient-free teacher policy to distill action spans. By unifying reinforcement learning, policy distillation, and executable action modeling, the proposed method achieves significant performance gains over state-of-the-art approaches on benchmark datasets including WebQSP, GrailQA, and GraphQ.

Abstract

Reinforcement learning (RL) is a natural fit for agentic knowledge base question answering (KBQA), where a model must issue executable actions, observe knowledge-base feedback, and eventually return an answer. However, current RL-based KBQA systems mainly optimize sparse rewards from the final answer, leaving intermediate action errors weakly supervised. This is especially limiting for logical-form annotated KBQA benchmarks: gold logical forms can be converted into executable action sequences, but existing pipelines use them mainly for warm-start data construction rather than for on-policy RL updates. We propose GAPD, a training-time Gold-Action Policy Distillation framework that adds dense token-level guidance to outcome-based RL. To align gold actions with on-policy student rollouts, GAPD uses MID-ANCHOR MATCHING: it treats the intermediate entities reached during student exploration and gold execution as state anchors, and matches student states to gold states through these explored entity sets. The current policy conditioned on this aligned gold action serves as a stop-gradient teacher, whose token distribution is distilled back to the ordinary student policy over generated action-token spans. GAPD consistently surpasses the current state of the art on WebQSP, GrailQA, and GraphQ.

Problem

• Knowledge Base Question Answering
• Reinforcement Learning
• Policy Distillation
• Sparse Rewards
• Logical Form

Innovation

• Policy Distillation
• Reinforcement Learning
• Knowledge Base Question Answering
• Intermediate Supervision
• State Alignment