Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination

2024-09-19

arXiv.org

AI Summary

Existing AI model evaluations inadequately assess commonsense reasoning and cross-lingual understanding—particularly on high-difficulty, exam-style multiple-choice questions (e.g., university entrance exams). Method: We introduce UNED-ACCESS 2024, a bilingual (Spanish–English), expert-translated, zero-publicly-contaminated benchmark comprising 1,003 aligned items. Under zero-shot evaluation, we systematically assess mainstream LMs’ cross-lingual knowledge reasoning capabilities, incorporating bilingual consistency analysis and Pearson correlation testing. Results: The top-performing models exhibit negligible performance gaps (<1%) between Spanish and English, whereas smaller models show up to a 37% gap. UNED-ACCESS correlates strongly with MMLU in model rankings (r = 0.98), validating its efficacy as a lightweight, subject-specific, high-fidelity alternative benchmark. This work provides the first empirical evidence that high-quality, small-scale bilingual exam item sets can efficiently capture general knowledge competence in LMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In this article we present UNED-ACCESS 2024, a bilingual dataset that consists of 1003 multiple-choice questions of university entrance level exams in Spanish and English. Questions are originally formulated in Spanish and translated manually into English, and have not ever been publicly released. A selection of current open-source and proprietary models are evaluated in a uniform zero-shot experimental setting both on the UNED-ACCESS 2024 dataset and on an equivalent subset of MMLU questions. Results show that (i) reasoning questions are challenging for models, (ii) smaller models perform worse than larger models and degrade faster in Spanish than in English and (iii) the performance gap between languages is negligible for the best models and grows up to 37% for smaller models. Model ranking on UNED-ACCESS 2024 is almost identical in English and Spanish, and has also a high correlation (0.98 Pearson) with ranking on MMLU, suggesting that a small dataset is sufficiently diverse and representative to measure performance by discipline.

Problem

• AI Model Evaluation
• Multilingual Understanding
• Commonsense Reasoning

Innovation

• Bilingual Benchmark
• Common Sense Reasoning
• Cross-lingual Performance Analysis