Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation

2026-04-13

AI Summary

This study addresses the limitations of existing multilingual synthetic data approaches, which overly rely on large teacher models while neglecting cross-lingual capability disparities, resulting in poor data quality and constrained student model performance. The authors systematically evaluate the effectiveness of diverse language models as teachers, generating 1.4 million supervised fine-tuning examples and training 240 student models. They propose the Polyglot Score—a holistic metric integrating intrinsic data quality and extrinsic student performance—and find that model scale is not a decisive factor; instead, attributes such as prompt diversity, response length, and fluency explain over 93.3% of the variance in data quality. The work further identifies effective strategies like teacher–student model family alignment and demonstrates that Gemma 3 27B and Aya Expanse 32B serve as robust, cross-family teachers, significantly enhancing performance on low-resource languages.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Synthesizing supervised finetuning (SFT) data from language models (LMs) to teach smaller models multilingual tasks has become increasingly common. However, teacher model selection is often ad hoc, typically defaulting to the largest available option, even though such models may have significant capability gaps in non-English languages. This practice can result in poor-quality synthetic data and suboptimal student downstream performance. In this work, we systematically characterize what makes an effective multilingual teacher. We measure intrinsic measures of data quality with extrinsic student model performance in a metric we call Polyglot Score; evaluating 10 LMs across 6 typologically diverse languages, generating over 1.4M SFT examples and training 240 student models. Among the models tested, Gemma 3 27B and Aya Expanse 32B emerge as consistently effective teachers across different student base model families. Further analyses reveal that model scale alone does not significantly predict teacher effectiveness; instead, data qualities such as prompt diversity, length, and response fluency capture over 93.3% of variance in intrinsic data quality and predict student performance. Finally, we provide practical recommendations, including matching the model families of teacher-student pairs and translating from or responding to existing prompts, which can yield improvements for less-resourced languages. We hope that our work advances data-centric research in multilingual synthetic data and LM development.

Problem

• multilingual synthetic data
• teacher model selection
• supervised finetuning
• language models
• data quality

Innovation

• multilingual synthetic data
• teacher-student alignment
• Polyglot Score
• data quality metrics
• language model evaluation