Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation

2026-08-25

AI Summary

本文探讨了多语言模型评价中的公平比较问题，通过控制变量实验揭示常用指标存在的偏差，并提出基于语义等价序列的句子级负对数似然作为更有效的跨语言比较方法。

Abstract

Crosslingual evaluation of language models that enables fair comparisons remains a fundamental challenge in multilingual NLP. Existing studies adopt a variety of downstream tasks and intrinsic metrics with different theoretical justifications, yet there has been little empirical investigation into whether these approaches yield meaningful crosslingual conclusions. We systematically examine crosslingual evaluation approaches using controlled monolingual language models trained on parallel data with varying tokenizer vocabulary sizes and model sizes, and further validate our findings on multilingual LLMs. We further discuss challenges in achieving comparable downstream evaluation across languages. Our results show that several widely used normalized metrics introduce crosslinguistic biases rooted in tokenization, encoding, and orthographic differences. In contrast, sentence-level negative log-likelihood computed over semantically equivalent sequences provides more meaningful and consistent crosslingual comparisons.

Problem

• Crosslingual Evaluation
• Language Models
• Fair Comparisons
• Multilingual NLP
• Normalized Metrics

Innovation

• crosslingual evaluation
• normalized metrics
• negative log-likelihood
• multilingual LLMs