KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering

2025-12-10

AI Summary

To address the propensity of large language models (LLMs) in knowledge base question answering (KBQA) to generate spurious queries and rely on rigid, context-agnostic templates, this paper reformulates the natural language-to-logical-form mapping as a multi-turn interactive decision process and introduces the first reinforcement learning framework guided by execution feedback. We propose Referenced Rejection Sampling (RRS), a data synthesis method that ensures reasoning trajectories strictly align with executable action sequences over the knowledge base, mitigating cold-start and hallucination issues. Integrated with Group Relative Policy Optimization (GRPO) and executable logical form generation, our approach enables robust multi-step knowledge graph navigation. Evaluated on WebQSP, GrailQA, and GraphQuestions, our method achieves state-of-the-art performance: logical form accuracy and execution verifiability significantly improve, hallucination rates decrease, and generalization to compositional and zero-shot reasoning is enhanced.

Technology Category

• Knowledge Representation and Reasoning: Knowledge Representation Languages
• Natural Language Processing: Question Answering
• Machine Learning: Statistical Relational/Logic Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Knowledge Base Question Answering (KBQA) challenges models to bridge the gap between natural language and strict knowledge graph schemas by generating executable logical forms. While Large Language Models (LLMs) have advanced this field, current approaches often struggle with a dichotomy of failure: they either generate hallucinated queries without verifying schema existence or exhibit rigid, template-based reasoning that mimics synthesized traces without true comprehension of the environment. To address these limitations, we present extbf{KBQA-R1}, a framework that shifts the paradigm from text imitation to interaction optimization via Reinforcement Learning. Treating KBQA as a multi-turn decision process, our model learns to navigate the knowledge base using a list of actions, leveraging Group Relative Policy Optimization (GRPO) to refine its strategies based on concrete execution feedback rather than static supervision. Furthermore, we introduce extbf{Referenced Rejection Sampling (RRS)}, a data synthesis method that resolves cold-start challenges by strictly aligning reasoning traces with ground-truth action sequences. Extensive experiments on WebQSP, GrailQA, and GraphQuestions demonstrate that KBQA-R1 achieves state-of-the-art performance, effectively grounding LLM reasoning in verifiable execution.

Problem

• Addresses hallucinated queries and rigid reasoning in KBQA
• Shifts from text imitation to interaction optimization via RL
• Resolves cold-start by aligning reasoning with ground-truth actions

Innovation

• Reinforcement Learning optimizes KBQA via interaction feedback
• Group Relative Policy Optimization refines strategies with execution results
• Referenced Rejection Sampling aligns reasoning with ground-truth actions