XIFBench: Evaluating Large Language Models on Multilingual Instruction Following

2025-03-10

AI Summary

Existing multilingual instruction-following evaluations for large language models (LLMs) lack fine-grained constraint analysis, hindering accurate assessment of their capabilities in low-resource languages. To address this, we propose XIFBench—the first constraint-driven benchmark for multilingual instruction following—covering six low-resource languages, 465 parallel instructions, and five novel constraint categories (e.g., cultural adaptation, logical consistency). We introduce an English semantic anchor–driven cross-lingual verification protocol and develop a comprehensive evaluation framework integrating a constraint taxonomy, parallel instruction construction, and multidimensional attribution analysis. Empirical results demonstrate a significant performance degradation of LLMs in low-resource languages, with constraint type, instruction complexity, and cultural specificity identified as key determinants. XIFBench enables systematic, interpretable, and culturally grounded evaluation of multilingual instruction following, advancing robustness and fairness assessment across linguistically diverse settings.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have demonstrated remarkable instruction-following capabilities across various applications. However, their performance in multilingual settings remains poorly understood, as existing evaluations lack fine-grained constraint analysis. We introduce XIFBench, a comprehensive constraint-based benchmark for assessing multilingual instruction-following abilities of LLMs, featuring a novel taxonomy of five constraint categories and 465 parallel instructions across six languages spanning different resource levels. To ensure consistent cross-lingual evaluation, we develop a requirement-based protocol that leverages English requirements as semantic anchors. These requirements are then used to validate the translations across languages. Extensive experiments with various LLMs reveal notable variations in instruction-following performance across resource levels, identifying key influencing factors such as constraint categories, instruction complexity, and cultural specificity.

Problem

• Evaluates multilingual instruction-following in LLMs.
• Introduces XIFBench for fine-grained constraint analysis.
• Assesses performance across languages and resource levels.

Innovation

• XIFBench: multilingual instruction-following benchmark
• Five constraint categories, 465 parallel instructions
• Requirement-based protocol for cross-lingual evaluation