LLMs Get Smarter from Targeted Synthetic Multilingual Data

2026-08-16

AI Summary

This study addresses the challenges of insufficient cross-lingual semantic alignment and capability disparities in large language models by proposing Hotfixr, a data-centric framework. This approach precisely identifies multilingual vulnerabilities and generates targeted synthetic data for fine-tuning, thereby effectively enhancing cross-lingual reasoning capabilities and robustness. Experimental results demonstrate that Hotfixr improves in-distribution performance by 6.2% and out-of-distribution language task accuracy by 7.1%, while reducing catastrophic forgetting by 3.7%. These findings indicate significant improvements in generalization and stability for large language models operating in low-resource scenarios, offering a viable solution to mitigate performance gaps across diverse linguistic contexts through strategic data augmentation and optimization.

Abstract

Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt. In other words, a language model outputs different (and potentially incorrect) responses to the same semantic query when prompted in different languages. Prior work attributes this to an internal misalignment of semantic representation across languages. Currently, there are two main approaches to address LSC in the literature: (1) routing all queries through English, improving performance, but limiting language expressivity to English; or (2) training on language-balanced data, equalizing model performance across languages, but reducing overall performance. In this work, we take a data centric perspective and introduce HOTFIXR: Hardness Optimized Training data For Improving X-Lingual Reasoning. It is a data generation framework that uses models to probe and learn a student model's multilingual weaknesses, and generates data to mitigate them. HOTFIXR can generate multilingual synthetic training data that can improve multilingual performance. We evaluate on three in-distribution tasks, three out-of-distribution tasks, and four out-of-distribution languages. On average, HOTFIXR (1) improves in-distribution performance by 6.2%, (2) reduces catastrophic forgetting (induced by fine-tuning) on OOD tasks by 3.7%, and (3) on OOD languages by 7.1%. Overall, as many real-world applications requires multilingual LLMs, our work contributes to the efforts of making LLMs multilingually proficient. We will release code upon acceptance.

Problem

• Language-specific competency
• Multilingual LLMs
• Semantic misalignment
• Cross-lingual reasoning

Innovation

• HOTFIXR
• Synthetic Multilingual Data
• Language-Specific Competency
• Data-Centric AI
• Cross-Lingual Reasoning