Facts Do Care About Your Language: Assessing Answer Quality of Multilingual LLMs

2025-06-03

AI Summary

This study investigates cross-lingual factual consistency of multilingual large language models (LLMs)—exemplified by Llama-3.1—in secondary-school-level factual question answering, focusing on the relationship between factual accuracy degradation and language rarity in non-English languages. Method: We introduce the first education-oriented multilingual factuality evaluation framework, comprising a manually verified multilingual benchmark dataset, bias quantification analysis, and controlled prompt ablation experiments. Contribution/Results: We empirically demonstrate a significant negative correlation between model factual accuracy and language rarity: average accuracy drops by 37% across 12 non-English languages, with error rates reaching 62% for low-resource languages. Existing multilingual alignment techniques fail to ensure factual reliability in such languages. These findings provide critical empirical evidence and an evaluative paradigm for advancing fairness and trustworthiness in educational AI systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Factuality is a necessary precursor to useful educational tools. As adoption of Large Language Models (LLMs) in education continues of grow, ensuring correctness in all settings is paramount. Despite their strong English capabilities, LLM performance in other languages is largely untested. In this work, we evaluate the correctness of the Llama3.1 family of models in answering factual questions appropriate for middle and high school students. We demonstrate that LLMs not only provide extraneous and less truthful information, but also exacerbate existing biases against rare languages.

Problem

• Assessing factuality of multilingual LLMs in education
• Evaluating LLM correctness in non-English languages
• Identifying biases and inaccuracies in rare language outputs

Innovation

• Evaluate Llama3.1 models' multilingual factuality
• Assess correctness for middle-high school questions
• Identify biases and extraneous information issues