When CQs Go Wrong: Challenges in CQ Verification with OE-Assist

2026-06-23

AI Summary

This study addresses the challenges posed by ambiguity and complexity in natural language competency questions (CQs), which often lead to inconsistencies in ontology modeling and validation errors. For the first time, it systematically analyzes the key factors contributing to these issues and proposes preprocessing CQs with tool support prior to publication to reduce ambiguity in subsequent engineering phases. To this end, the authors developed OE-Assist, an ontology evaluation assistance platform powered by large language models (LLMs), complemented by user behavior analysis methods. An empirical evaluation involving 19 participants completing 20 tasks demonstrates that CQ preprocessing significantly improves validation consistency and user performance, thereby confirming the effectiveness and necessity of LLM-assisted support in ontology evaluation.

Abstract

Competency Questions (CQs) are the central component of CQ-verification, an established process in which an ontology is evaluated against a set of natural language questions to determine whether the intended purpose of the ontology has been properly modelled. However, CQ-verification is often time-consuming and error-prone, as it requires careful interpretation of linguistic nuances and precise alignment with formal ontology constructs. Ambiguities and complexity in CQs can further complicate this process, leading to inconsistent modelling decisions and verification outcomes. In this paper, we investigate what makes a CQ challenging and possible solutions to enhance the users' performance in the CQ-verification process. We experimented with the data of 19 participants who performed CQ-verification on 20 tasks using an LLM assistant to support ontology evaluation. The results show the necessity of a tool to refine CQs before publishing them to avoid ambiguity or excessive complexity in later phases of the ontology engineering process.

Problem

• Competency Questions
• CQ verification
• ontology engineering
• ambiguity
• complexity

Innovation

• Competency Questions
• CQ verification
• ontology engineering
• LLM assistant
• natural language ambiguity