Q${}^2$Forge: Minting Competency Questions and SPARQL Queries for Question-Answering Over Knowledge Graphs

2025-05-19

AI Summary

Knowledge graphs (KGs) face usability bottlenecks: non-expert users struggle to formulate SPARQL queries; canonical questions (CQs) and example queries are scarce and low-quality; and high-fidelity NL-SPARQL benchmark datasets are lacking. Method: We propose the first end-to-end human-in-the-loop CQ–SPARQL generation framework, integrating LLM-driven prompt engineering, syntax-constrained SPARQL generation, and a modular pipeline (CQ generation → query generation → refinement). Crucially, we introduce an iterative validation mechanism wherein an LLM acts as an intelligent adjudicator, guided by human feedback for continuous optimization. Contribution/Results: We release an open, extensible architecture enabling semantic-accurate, KG-agnostic CQ and corresponding SPARQL generation. Our framework significantly improves KG documentation efficiency, produces high-quality benchmark query sets, and establishes a new paradigm for enhancing KG accessibility and training NL2SPARQL models.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Natural Language Processing: Question Answering
• Knowledge Representation and Reasoning: Description Logics

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The SPARQL query language is the standard method to access knowledge graphs (KGs). However, formulating SPARQL queries is a significant challenge for non-expert users, and remains time-consuming for the experienced ones. Best practices recommend to document KGs with competency questions and example queries to contextualise the knowledge they contain and illustrate their potential applications. In practice, however, this is either not the case or the examples are provided in limited numbers. Large Language Models (LLMs) are being used in conversational agents and are proving to be an attractive solution with a wide range of applications, from simple question-answering about common knowledge to generating code in a targeted programming language. However, training and testing these models to produce high quality SPARQL queries from natural language questions requires substantial datasets of question-query pairs. In this paper, we present Q${}^2$Forge that addresses the challenge of generating new competency questions for a KG and corresponding SPARQL queries. It iteratively validates those queries with human feedback and LLM as a judge. Q${}^2$Forge is open source, generic, extensible and modular, meaning that the different modules of the application (CQ generation, query generation and query refinement) can be used separately, as an integrated pipeline, or replaced by alternative services. The result is a complete pipeline from competency question formulation to query evaluation, supporting the creation of reference query sets for any target KG.

Problem

• Generating SPARQL queries for non-expert KG users
• Creating competency questions for knowledge graphs
• Automating question-query pair dataset production

Innovation

• Generates competency questions and SPARQL queries
• Uses LLMs and human feedback for validation
• Modular pipeline for KG query generation