Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgish

2025-04-02

AI Summary

Reliable evaluation frameworks for large language models (LLMs) in low-resource languages—such as Luxembourgish—remain scarce. Method: This work introduces, for the first time, standardized language proficiency examinations as a zero-shot and few-shot evaluation framework for LLMs in under-resourced languages. We benchmark multiple models—including ChatGPT, Claude, and DeepSeek-R1—on Luxembourgish-language exams. Contribution/Results: We find a strong positive correlation between model scale and performance: large models achieve mean scores >80%, while small models score <40%. Crucially, exam scores exhibit high predictive validity for downstream NLP tasks, correlating strongly (r > 0.85) with F1 scores on named entity recognition (NER) and part-of-speech (POS) tagging. This approach bridges a critical gap in systematic LLM evaluation for low-resource languages and establishes a reusable, interpretable paradigm for characterizing linguistic capability in minority-language models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have become an increasingly important tool in research and society at large. While LLMs are regularly used all over the world by experts and lay-people alike, they are predominantly developed with English-speaking users in mind, performing well in English and other wide-spread languages while less-resourced languages such as Luxembourgish are seen as a lower priority. This lack of attention is also reflected in the sparsity of available evaluation tools and datasets. In this study, we investigate the viability of language proficiency exams as such evaluation tools for the Luxembourgish language. We find that large models such as ChatGPT, Claude and DeepSeek-R1 typically achieve high scores, while smaller models show weak performances. We also find that the performances in such language exams can be used to predict performances in other NLP tasks.

Problem

• Evaluating LLM performance in low-resource languages like Luxembourgish
• Assessing language proficiency exams as evaluation tools for Luxembourgish
• Predicting NLP task performance using language exam results

Innovation

• Using language proficiency exams for evaluation
• Testing LLMs on low-resource Luxembourgish language
• Predicting NLP task performance via exam results