A Set of Quebec-French Corpus of Regional Expressions and Terms

2025-10-06

AI Summary

Existing NLP evaluation benchmarks lack standardized, high-quality resources for assessing large language models’ (LLMs) comprehension of regionally specific linguistic expressions—particularly in low-resource dialects such as Quebec French. Method: We introduce two rigorously constructed, reproducible benchmark datasets—QFrCoRE (4,633 regionally attested idioms) and QFrCoRT (171 regionally marked lexical items)—and propose a novel “idiom-as-probe” paradigm for dialect capability assessment. Leveraging systematic corpus collection and expert annotation, we conduct empirical evaluation across 94 LLMs. Contribution/Results: Our benchmarks effectively discriminate fine-grained semantic understanding of Quebec French, revealing substantial performance gaps across models. They provide the first reliable, granular evaluation framework for dialectal adaptation, addressing a critical gap in low-resource dialect comprehension assessment and enabling targeted development of geographically aware language models.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

The tasks of idiom understanding and dialect understanding are both well-established benchmarks in natural language processing. In this paper, we propose combining them, and using regional idioms as a test of dialect understanding. Towards this end, we propose two new benchmark datasets for the Quebec dialect of French: QFrCoRE, which contains 4,633 instances of idiomatic phrases, and QFrCoRT, which comprises 171 regional instances of idiomatic words. We explain how to construct these corpora, so that our methodology can be replicated for other dialects. Our experiments with 94 LLM demonstrate that our regional idiom benchmarks are a reliable tool for measuring a model's proficiency in a specific dialect.

Problem

• Creating benchmark datasets for Quebec-French dialect understanding
• Evaluating LLM proficiency using regional idiom comprehension tests
• Developing reproducible methodology for dialect-specific corpus construction

Innovation

• Constructed two Quebec-French idiom benchmark datasets
• Proposed methodology for replicating corpora across dialects
• Used regional idioms to test dialect understanding in LLMs