CroissantLLM: A Truly Bilingual French-English Language Model

2024-02-01

arXiv.org

AI Summary

Language model research remains heavily English-centric, with a scarcity of high-performance, open-source bilingual (French–English) models. Method: We introduce the first truly bilingual 1.3B-parameter large language model, pretrained on a balanced 3-terabyte English–French corpus using a novel 1:1 data-ratio paradigm; we design a custom bilingual tokenizer and employ multi-stage bilingual fine-tuning. Contribution/Results: We release FrenchBench—the first comprehensive, French-specific evaluation benchmark—and achieve 81% FMTI transparency, substantially exceeding mainstream open-source projects. The model enables efficient deployment on consumer-grade hardware and significantly outperforms same-scale monolingual baselines on FrenchBench. We fully open-source all training data—including a human-curated French subset—training code, dozens of intermediate and final checkpoints, and high-performance fine-tuned variants for dialogue and machine translation.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local hardware. To that end, we pioneer the approach of training an intrinsically bilingual model with a 1:1 English-to-French pretraining data ratio, a custom tokenizer, and bilingual finetuning datasets. We release the training dataset, notably containing a French split with manually curated, high-quality, and varied data sources. To assess performance outside of English, we craft a novel benchmark, FrenchBench, consisting of an array of classification and generation tasks, covering various orthogonal aspects of model performance in the French Language. Additionally, rooted in transparency and to foster further Large Language Model research, we release codebases, and dozens of checkpoints across various model sizes, training data distributions, and training steps, as well as fine-tuned Chat models, and strong translation models. We evaluate our model through the FMTI framework, and validate 81 % of the transparency criteria, far beyond the scores of even most open initiatives. This work enriches the NLP landscape, breaking away from previous English-centric work in order to strengthen our understanding of multilinguality in language models.

Problem

• Develop a bilingual French-English language model
• Create high-quality open-source bilingual training data
• Establish benchmarks for French language model performance

Innovation

• Bilingual training with 1:1 English-French ratio
• Custom tokenizer and bilingual finetuning datasets
• Novel French benchmark for model evaluation