MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

2025-04-03

AI Summary

Existing large language models (LLMs) lack rigorous evaluation of syntactic competence—particularly morphosyntactic judgment—in low-resource languages. Method: We introduce MorphoGram, the first multilingual syntactic benchmark covering 101 languages and six core grammatical phenomena, comprising 125,000 minimal pairs. Built upon Universal Dependencies and UniMorph, MorphoGram employs an automated, rule-driven morphological generation pipeline for scalable, cross-lingual construction. Contribution/Results: Our systematic evaluation reveals a pronounced performance drop in syntactic judgment for mainstream LLMs on low-resource languages, demonstrating strong resource dependency. MorphoGram fills a critical gap in fine-grained, multilingual grammatical assessment—spanning over one hundred languages—and establishes a reproducible, extensible evaluation paradigm with empirical grounding for low-resource language modeling.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages, 6 linguistic phenomena and containing more than 125,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.

Problem

• Evaluates LLMs' abilities across 101 languages
• Assesses 6 linguistic phenomena using minimal pairs
• Highlights shortcomings in low-resource language modeling

Innovation

• Automated pipeline for minimal pairs creation
• Leverages Universal Dependencies and UniMorph
• Benchmark covering 101 languages