Evaluating the Fitness of Ontologies for the Task of Question Generation

2025-04-08

AI Summary

This study investigates the applicability of ontologies to automated question generation (AQG) in educational settings, focusing on identifying ontology features—such as structural complexity, semantic richness, and concept coverage—that critically influence question quality and cognitive difficulty. We propose the first ontology suitability assessment framework tailored for AQG, integrating the ROMEO methodology, domain expert evaluation, and empirical comparison across multiple ontologies to define task-oriented requirements and quantifiable metrics. Experimental results demonstrate that ontology characteristics significantly affect AQG performance, with substantial variation observed across different ontologies. This work bridges a key research gap by establishing the linkage between ontology quality assessment and AQG effectiveness, thereby providing both theoretical foundations and practical guidelines for knowledge graph–driven intelligent item generation in education.

Technology Category

• Knowledge Representation and Reasoning: Ontologies
• Data Mining & Knowledge Management: Intelligent Query Processing
• Natural Language Processing: Question Answering

Application Category

• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Community question answering

Abstract

Ontology-based question generation is an important application of semantic-aware systems that enables the creation of large question banks for diverse learning environments. The effectiveness of these systems, both in terms of the calibre and cognitive difficulty of the resulting questions, depends heavily on the quality and modelling approach of the underlying ontologies, making it crucial to assess their fitness for this task. To date, there has been no comprehensive investigation into the specific ontology aspects or characteristics that affect the question generation process. Therefore, this paper proposes a set of requirements and task-specific metrics for evaluating the fitness of ontologies for question generation tasks in pedagogical settings. Using the ROMEO methodology, a structured framework for deriving task-specific metrics, an expert-based approach is employed to assess the performance of various ontologies in Automatic Question Generation (AQG) tasks, which is then evaluated over a set of ontologies. Our results demonstrate that ontology characteristics significantly impact the effectiveness of question generation, with different ontologies exhibiting varying performance levels. This highlights the importance of assessing ontology quality with respect to AQG tasks.

Problem

• Evaluating ontology fitness for question generation tasks
• Identifying key ontology characteristics affecting question quality
• Proposing metrics to assess ontology performance in AQG

Innovation

• Proposes task-specific metrics for ontology evaluation
• Uses ROMEO methodology for structured metric derivation
• Expert-based approach assesses ontology performance in AQG