TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representation

2026-03-09

AI Summary

This work addresses the poor performance of large language models on most low-resource European languages, primarily caused by severe training data bias toward high-resource languages such as English. To mitigate this imbalance, the authors develop a 30-billion-parameter open-source large language model supporting 34 European languages and introduce a novel training paradigm that integrates curriculum learning with dynamic data upsampling. During pretraining, the model alternates between uniform and natural language data distributions to enhance representation learning for underrepresented language families—including Baltic, Uralic, and Slavic languages—without increasing model size or total training compute. The approach substantially improves cross-lingual performance parity, outperforming existing open-source models across multiple multilingual benchmarks. Human evaluations reveal up to a tenfold reduction in linguistic errors, and the code, model weights, and training pipeline have been publicly released on Hugging Face.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Large language models often underperform in many European languages due to the dominance of English and a few high-resource languages in training data. This paper presents TildeOpen LLM, a 30-billion-parameter open-weight foundational model trained for 34 European languages to promote linguistic equity and improve performance for low-resource languages. To address the data imbalance, we combine dataset upsampling with a curriculum-based training schedule that alternates between uniform and natural language distributions. The resulting model performs favorably compared to other multilingual LLMs despite being trained with significantly fewer computing resources. Evaluation across multiple multilingual benchmarks shows that TildeOpen surpasses existing open-weight models in text generation and comprehension, particularly for Baltic, Finno-Ugric, and Slavic languages. Human evaluations confirm an up to tenfold reduction in linguistic errors relative to leading baselines. The model and associated resources are fully open-weight and publicly available at huggingface.co/TildeAI/TildeOpen-30b. These outcomes demonstrate that careful data curation and balanced training strategies can substantially enhance multilingual model quality without increasing model size or training volume.

Problem

• linguistic equity
• low-resource languages
• data imbalance
• multilingual LLMs
• language representation

Innovation

• Curriculum Learning
• Linguistic Equity
• Low-resource Languages
• Multilingual LLM
• Data Upsampling