BioGraphletQA: Knowledge-Anchored Generation of Complex QA Datasets

2026-04-28

AI Summary

This study addresses the scarcity of high-quality, complex, and factually verifiable question-answering datasets in the biomedical domain. To this end, the authors propose a knowledge graphlet-anchored generation framework that leverages small subgraphs as controllable anchors to guide large language models—via structured prompt engineering—in generating question-answer pairs that are both complex and factually consistent. Accuracy is further ensured by aligning generated content with excerpts from PubMed literature. The resulting BioGraphletQA dataset comprises 119,856 question-answer pairs and demonstrates substantial performance gains under low-resource settings: accuracy on PubMedQA improves from 49.2% to 68.5%, and MedQA achieves 44.8%.

Abstract

This paper presents a principled and scalable framework for systematically generating complex Question Answering (QA) data. In the core of this framework is a graphlet-anchored generation process, where small subgraphs from a Knowledge Graph (KG) are used in a structured prompt to control the complexity and ensure the factual grounding of questions generated by Large Language Models. The first instantiation of this framework is BioGraphletQA, a new biomedical KGQA dataset of 119,856 QA pairs. Each entry is grounded in a graphlet of up to five nodes from the OREGANO KG, with most of the pairs being enriched with relevant document snippets from PubMed. We start by demonstrating the framework's value and the dataset's quality through evaluation by a domain expert on 106 QA pairs, confirming the high scientific validity and complexity of the generated data. Secondly, we establish its practical utility by showing that augmenting downstream benchmarks with our data improves accuracy on PubMedQA from 49.2% to 68.5% in a low-resource setting, and on MedQA from a 41.4% baseline to 44.8% in a full-resource setting. Our framework provides a robust and generalizable solution for creating critical resources to advance complex QA tasks, including MCQA and KGQA. All resources supporting this work, including the dataset (https://zenodo.org/records/17381119) and framework code (https://github.com/ieeta-pt/BioGraphletQA), are publicly available to facilitate use, reproducibility and extension.

Problem

• Question Answering
• Knowledge Graph
• Dataset Generation
• Biomedical QA
• Factual Grounding

Innovation

• graphlet-anchored generation
• Knowledge Graph QA
• factual grounding
• complex question answering
• biomedical dataset generation