COLE: a Comprehensive Benchmark for French Language Understanding Evaluation

2025-10-06

AI Summary

To address the lack of systematic, language-specific evaluation benchmarks for French natural language understanding (NLU), this work introduces COLE—the first comprehensive multilingual benchmark tailored to French linguistic phenomena, including inflectional morphology, clitic dropping, and regional variation. COLE comprises 23 classification, regression, and generation tasks spanning sentiment analysis, paraphrase identification, grammaticality judgment, and logical inference. It supports zero-shot, few-shot, and fully supervised evaluation protocols. We systematically assess 94 large language models (LLMs), revealing substantial performance gaps between proprietary and open-source models. Key bottlenecks are identified in zero-shot extractive question answering, fine-grained word sense disambiguation, and regional variant comprehension. The benchmark is publicly released, providing a standardized evaluation framework and concrete directions for advancing French NLU research.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

To address the need for a more comprehensive evaluation of French Natural Language Understanding (NLU), we introduce COLE, a new benchmark composed of 23 diverse task covering a broad range of NLU capabilities, including sentiment analysis, paraphrase detection, grammatical judgment, and reasoning, with a particular focus on linguistic phenomena relevant to the French language. We benchmark 94 large language models (LLM), providing an extensive analysis of the current state of French NLU. Our results highlight a significant performance gap between closed- and open-weights models and identify key challenging frontiers for current LLMs, such as zero-shot extractive question-answering (QA), fine-grained word sense disambiguation, and understanding of regional language variations. We release COLE as a public resource to foster further progress in French language modelling.

Problem

• Creating a comprehensive French NLU benchmark with 23 diverse tasks
• Evaluating 94 large language models on French language understanding capabilities
• Identifying performance gaps in French-specific linguistic phenomena and challenges

Innovation

• Introduced COLE benchmark with 23 diverse NLU tasks
• Evaluated 94 large language models for French understanding
• Identified performance gaps in zero-shot QA and disambiguation