LLäMmlein: Compact and Competitive German-Only Language Models from Scratch

2024-11-17

arXiv.org

AI Summary

A scarcity of dedicated large language models (LLMs) for German hinders progress in German NLP. Method: We train from scratch and open-source two lightweight, decoder-only German LLMs—LLämlein-120M and LLämlein-1B—alongside their full training corpus. We introduce an end-to-end training framework featuring a custom German BPE tokenizer, multi-stage data cleaning, and checkpoint-based progressive evaluation. Crucially, we integrate SuperGLEBer—a novel dynamic monitoring tool—for the first time in monolingual German modeling, enabling early detection of task performance saturation to guide resource-efficient training. Results: Both models match or surpass same-scale state-of-the-art models on the SuperGLEBer benchmark, demonstrating the competitiveness and reproducibility of compact monolingual LLMs. This work delivers a scalable, transparent, and easily iteratable infrastructure for the German NLP community.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

We create two German-only decoder models, LL""aMmlein 120M and 1B, transparently from scratch and publish them, along with the training data, for the German NLP research community to use. The model training involved several key steps, including extensive data preprocessing, the creation of a custom German tokenizer, the training itself, as well as the evaluation of the final models on various benchmarks. Throughout the training process, multiple checkpoints were saved and analyzed using the SuperGLEBer benchmark to monitor the models' learning dynamics. Compared to state-of-the-art models on the SuperGLEBer benchmark, both LL""aMmlein models performed competitively, consistently matching or surpassing models with similar parameter sizes. The results show that the models' quality scales with size as expected, but performance improvements on some tasks plateaued early, offering valuable insights into resource allocation for future model development.

Problem

• Developing German-only language models from scratch
• Evaluating model performance on German NLP benchmarks
• Analyzing resource allocation for optimal model scaling

Innovation

• Custom German tokenizer for model training
• Extensive data preprocessing before training
• SuperGLEBer benchmark for learning analysis