Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting

2025-10-23

AI Summary

Existing large language models (LLMs) lack rigorous, theory-grounded evaluation frameworks for grammatical competence in low-resource endangered languages such as Irish. Method: We introduce the first fine-grained grammatical evaluation benchmark for Irish, comprising 1,020 human-annotated minimal-pair sentences covering 11 core syntactic and morphological features. The dataset is grounded in linguistic theory and validated by native speakers, and grammatical acceptability is assessed via standardized syntactic judgment tasks. Contribution/Results: Human performance achieves 90.1% accuracy—significantly outperforming the best-performing LLM (GPT-5, 73.5%), revealing a 16.6-point gap. Closed-source models outperform open-source counterparts by 18.1%, with qualitatively distinct error patterns. This work establishes the first reproducible, linguistically motivated grammatical assessment framework for Irish, exposing fundamental limitations in current LLMs’ syntactic representation of low-resource languages.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present Irish-BLiMP (Irish Benchmark of Linguistic Minimal Pairs), the first dataset and framework designed for fine-grained evaluation of linguistic competence in the Irish language, an endangered language. Drawing on a variety of linguistic literature and grammar reference works, we manually constructed and reviewed 1020 minimal pairs across a taxonomy of 11 linguistic features, through a team of fluent Irish speakers. We evaluate both existing Large Language Models (LLMs) and fluent human participants on their syntactic knowledge of Irish. Our findings show that humans outperform all models across all linguistic features, achieving 16.6% higher accuracy on average. Moreover, a substantial performance gap of 18.1% persists between open- and closed-source LLMs, with even the strongest model (gpt-5) reaching only 73.5% accuracy compared to 90.1% by human. Interestingly, human participants and models struggle on different aspects of Irish grammar, thus highlighting a difference in representation learned by the models. Overall, Irish-BLiMP provides the first systematic framework for evaluating the grammatical competence of LLMs in Irish and offers a valuable benchmark for advancing research on linguistic understanding in low-resource languages.

Problem

• Evaluating linguistic competence in endangered Irish language
• Assessing syntactic knowledge gaps between humans and LLMs
• Establishing benchmark for low-resource language model evaluation

Innovation

• Manually constructed 1020 Irish minimal pairs
• Evaluated LLMs and humans on syntactic knowledge
• Provided first systematic Irish grammatical evaluation framework