Scaling, Simplification, and Adaptation: Lessons from Pretraining on Machine-Translated Text

2025-09-21

AI Summary

To address the data scarcity barrier for pretraining low-resource languages, this work investigates the feasibility of using machine translation (MT)-generated text as a substitute for native corpora, focusing on three key questions: (1) scalability of MT data with model size, (2) impact of source-side simplification (e.g., LLM rewriting) on generalization to native text, and (3) adaptability of MT-pretrained models to limited native data. We pretrain GPT-2 variants (124M–774M parameters) on English→Indonesian and English→Tamil MT data and systematically evaluate syntactic probing and downstream task performance. Results show that MT data enables effective scaling; source-side simplification harms generalization to native text; and fine-tuning on only a small amount of native data surpasses models trained exclusively on native data. This is the first systematic study to empirically validate the efficacy and transfer potential of MT data in large-scale multilingual pretraining, offering a novel pathway to mitigate the “multilingual curse.”

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Most languages lack sufficient data for large-scale monolingual pretraining, creating a "data wall." Multilingual pretraining helps but is limited by language imbalance and the "curse of multilinguality." An alternative is to translate high-resource text with machine translation (MT), which raises three questions: (1) How does MT-derived data scale with model capacity? (2) Can source-side transformations (e.g., simplifying English with an LLM) improve generalization to native text? (3) How well do models pretrained on MT-derived data adapt when continually trained on limited native text? We investigate these questions by translating English into Indonesian and Tamil--two typologically distant, lower-resource languages--and pretraining GPT-2 models (124M-774M) on native or MT-derived corpora from raw and LLM-simplified English. We evaluate cross-entropy loss on native text, along with accuracy on syntactic probes and downstream tasks. Our results show that (1) MT-pretrained models benefit from scaling; (2) source-side simplification harms generalization to native text; and (3) adapting MT-pretrained models on native text often yields better performance than native-only models, even with less native data. However, tasks requiring cultural nuance (e.g., toxicity detection) demand more exposure to native data.

Problem

• Scaling machine translation pretraining effects with model capacity
• Improving generalization using source-side text simplification techniques
• Adapting MT-pretrained models with limited native text data

Innovation

• Pretraining GPT-2 models on machine-translated text
• Scaling model capacity improves MT-pretrained performance
• Adapting MT-pretrained models with native text enhances results