QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments

2025-08-22

AI Summary

Existing multilingual large language models (LLMs), particularly cross-lingual Transformers, exhibit limited understanding of syntactic norms specific to regional French varieties, such as Quebec French. Method: We introduce QFrCoLA—the first linguistically grounded acceptability judgment benchmark for Quebec French—comprising 25,153 in-domain and 2,675 out-of-domain sentences, annotated strictly according to descriptive syntactic norms rather than subjective judgments. Using QFrCoLA, we evaluate multilingual LLMs under both fine-tuned and zero-shot settings, comparing performance against general-purpose multilingual binary acceptability corpora. Contribution/Results: Empirical results reveal a substantial performance gap for current cross-lingual models on QFrCoLA relative to other languages, indicating a critical deficiency in their internalization of Quebec French syntax. Thus, QFrCoLA serves as a rigorous, linguistically informed diagnostic benchmark for assessing syntactic competence in regional French varieties, addressing a key gap in evaluation resources for geographically specific French dialects.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Large and Transformer-based language models perform outstandingly in various downstream tasks. However, there is limited understanding regarding how these models internalize linguistic knowledge, so various linguistic benchmarks have recently been proposed to facilitate syntactic evaluation of language models across languages. This paper introduces QFrCoLA (Quebec-French Corpus of Linguistic Acceptability Judgments), a normative binary acceptability judgments dataset comprising 25,153 in-domain and 2,675 out-of-domain sentences. Our study leverages the QFrCoLA dataset and seven other linguistic binary acceptability judgment corpora to benchmark seven language models. The results demonstrate that, on average, fine-tuned Transformer-based LM are strong baselines for most languages and that zero-shot binary classification large language models perform poorly on the task. However, for the QFrCoLA benchmark, on average, a fine-tuned Transformer-based LM outperformed other methods tested. It also shows that pre-trained cross-lingual LLMs selected for our experimentation do not seem to have acquired linguistic judgment capabilities during their pre-training for Quebec French. Finally, our experiment results on QFrCoLA show that our dataset, built from examples that illustrate linguistic norms rather than speakers' feelings, is similar to linguistic acceptability judgment; it is a challenging dataset that can benchmark LM on their linguistic judgment capabilities.

Problem

• Evaluating linguistic knowledge internalization in large language models
• Assessing syntactic capabilities across multiple languages via benchmarks
• Measuring Quebec-French linguistic acceptability judgment performance in LLMs

Innovation

• Fine-tuned Transformer-based LM baseline
• Zero-shot LLMs perform poorly classification
• Cross-lingual LLMs lack Quebec French capability