FoQA: A Faroese Question-Answering Dataset

2025-02-11

AI Summary

Faroean lacks high-quality question answering (QA) datasets, hindering NLP advancement for this low-resource language. To address this, we introduce FoQA—the first extractive QA dataset for Faroese—constructed from Faroese Wikipedia. We employ GPT-4-turbo to generate initial QA pairs, augment question difficulty via paraphrasing, and conduct multi-round native-speaker validation, yielding 2,000 high-quality instances. Our methodology establishes a semi-automated “LLM-assisted + human refinement” paradigm for QA dataset construction, pioneering the first Faroese QA benchmark. We publicly release the development set, full dataset, and an error analysis subset. Comprehensive baseline evaluations on BERT-based models and state-of-the-art LLMs confirm FoQA’s validity and challenge level. All resources are fully open-sourced, filling a critical gap in QA research for under-resourced languages.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We present FoQA, a Faroese extractive question-answering (QA) dataset with 2,000 samples, created using a semi-automated approach combining Large Language Models (LLMs) and human validation. The dataset was generated from Faroese Wikipedia articles using GPT-4-turbo for initial QA generation, followed by question rephrasing to increase complexity and native speaker validation to ensure quality. We provide baseline performance metrics for FoQA across multiple models, including LLMs and BERT, demonstrating its effectiveness in evaluating Faroese QA performance. The dataset is released in three versions: a validated set of 2,000 samples, a complete set of all 10,001 generated samples, and a set of 2,395 rejected samples for error analysis.

Problem

• Faroese QA dataset creation
• LLM and human validation
• Baseline performance evaluation

Innovation

• LLMs for QA generation
• Human validation ensures quality
• Multiple dataset versions available