Bench4KE: Benchmarking Automated Competency Question Generation

2025-05-30

AI Summary

Automated knowledge engineering (KE) tools lack standardized evaluation, hindering methodological rigor and result reproducibility. Method: We introduce the first scalable API-based benchmark platform for LLM-driven ontology competency question (CQ) generation. It features the first gold-standard CQ dataset spanning four real-world ontology projects and a multidimensional evaluation framework integrating semantic similarity (BERTScore, ROUGE, BLEU), logical consistency, and diversity. The platform adopts a modular API architecture, enabling extensibility to downstream KE tasks such as SPARQL query generation. Contribution/Results: Using this platform, we conduct a comprehensive benchmark of four state-of-the-art LLM-CQ systems, establishing performance baselines. All code, datasets, and evaluation frameworks are publicly released under the Apache 2.0 license to advance standardization in automated KE evaluation.

Technology Category

• Knowledge Representation and Reasoning: Ontologies
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Natural Language Processing: Question Answering

Application Category

• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The availability of Large Language Models (LLMs) presents a unique opportunity to reinvigorate research on Knowledge Engineering (KE) automation, a trend already evident in recent efforts developing LLM-based methods and tools for the automatic generation of Competency Questions (CQs). However, the evaluation of these tools lacks standardisation. This undermines the methodological rigour and hinders the replication and comparison of results. To address this gap, we introduce Bench4KE, an extensible API-based benchmarking system for KE automation. Its first release focuses on evaluating tools that generate CQs automatically. CQs are natural language questions used by ontology engineers to define the functional requirements of an ontology. Bench4KE provides a curated gold standard consisting of CQ datasets from four real-world ontology projects. It uses a suite of similarity metrics to assess the quality of the CQs generated. We present a comparative analysis of four recent CQ generation systems, which are based on LLMs, establishing a baseline for future research. Bench4KE is also designed to accommodate additional KE automation tasks, such as SPARQL query generation, ontology testing and drafting. Code and datasets are publicly available under the Apache 2.0 license.

Problem

• Lack of standardized evaluation for LLM-based CQ generation tools
• Need for rigorous benchmarking in Knowledge Engineering automation
• Absence of comparative baselines for Competency Question generation systems

Innovation

• API-based benchmarking system for KE automation
• Uses similarity metrics to assess CQ quality
• Accommodates additional KE automation tasks