Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability

2026-06-20

AI Summary

This study systematically evaluates the machine translation performance of large language models on English-to-Hausa and English-to-Fon—two low-resource West African languages—and assesses the reliability of automatic evaluation metrics against human judgments. Using multi-scale datasets ranging from 500 to 10,000 sentences, the authors combine automatic metrics (BLEU, chrF++, TER, COMET, and BERTScore) with native-speaker human ratings. Results reveal high translation quality for Hausa (human scores of 4.0–4.5/5) but substantially poorer performance for Fon (1.0–2.2/5), with no cross-lingual transferability between the two. Neural metrics exhibit unstable correlations with human judgments and suffer from embedding collapse—a phenomenon this work identifies for the first time in this context. The authors propose a multi-metric ensemble strategy and recommend a minimum of 2,500 sentence pairs to achieve stable system rankings.

Abstract

We investigate the translation quality of current large language models (LLMs) for English-to-Hausa and English-to-Fongbe - two typologically distinct West African languages from the Afroasiatic and Niger-Congo families respectively - and evaluate whether standard automatic metrics reliably reflect human judgment for these low-resource languages. We evaluate four models (GPT-4o Mini, Claude Sonnet 4, Gemini 2.5 Flash, and Qwen2.5-7B) at progressive scales (500 to 10,000 sentences) using automatic metrics (BLEU, chrF++, TER, COMET, BERTScore) validated against native-speaker judgment. Our results reveal three key findings. First, translation quality varies substantially by language: Hausa achieves acceptable quality (human scores 4.0-4.5/5) while Fongbe achieves poor quality (1.0-2.2/5), with a consistent 3x BLEU gap across all systems. Second, model rankings differ by language - Gemini leads for Fongbe while GPT-4o leads for Hausa by human evaluation - indicating that performance on one low-resource African language does not predict performance on another. Third, metric-human correlation varies dramatically: perfect rank correlation for Fongbe (rho=1.0) but weak correlation for Hausa (rho=0.5), where human evaluators preferred GPT-4o despite all automatic metrics ranking Claude first. We further show that neural metrics like BERTScore exhibit embedding collapse (within-language similarity >0.99) for both languages, limiting their ability to differentiate translation quality. Based on these findings, we recommend multi-metric evaluation for low-resource African languages, with particular caution when interpreting neural metrics. We establish that minimum sample sizes of n=2,500 sentences are required for stable system rankings, as smaller samples produced artifact findings that reversed at scale.

Problem

• machine translation
• low-resource languages
• evaluation metrics
• Hausa
• Fongbe

Innovation

• low-resource languages
• metric reliability
• embedding collapse
• human evaluation
• machine translation benchmark