Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code

2024-03-30

AI Summary

To address catastrophic forgetting, insufficient safety alignment, high computational costs, and weak regulatory compliance in multilingual large language models (LLMs) under continual learning, this work introduces Aurora-M, a 15B-parameter open-source multilingual LLM. Built upon the StarCoderPlus architecture, it undergoes continual pretraining on 435B new tokens—exceeding 2T total tokens—incorporating English, Finnish, Hindi, Japanese, Vietnamese, and multilingual code corpora. We propose the first human-audited safety instruction tuning paradigm, explicitly designed to fulfill the safety, trustworthiness, and governance requirements of the U.S. Biden-Harris Executive Order on AI; it further integrates red-teaming and regulatory compliance validation. Experiments demonstrate substantial improvements over baselines in multilingual understanding, code generation, and safety evaluation, alongside robust retention under continual learning. The model and its variants are publicly released on Hugging Face.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Pretrained language models are an integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim to democratize access to pretrained models for collaborative community development. Despite these efforts, such models encounter challenges such as limited multilingual capabilities, risks of catastrophic forgetting during continual pretraining, and the high costs of training models from scratch, alongside the need to align with AI safety standards and regulatory frameworks. This paper presents Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. Continually pretrained from StarCoderPlus on 435B additional tokens, Aurora-M surpasses 2T tokens in total training token count. It is the first open-source multilingual model fine-tuned on human-reviewed safety instructions, thus aligning its development not only with conventional red-teaming considerations, but also with the specific concerns articulated in the Biden-Harris Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. We evaluate Aurora-M across a wide range of tasks and languages, showcasing its robustness against catastrophic forgetting and its superior performance in multilingual settings, particularly in safety evaluations. We open-source Aurora-M and its variants to encourage responsible open-source development of large language models at https://huggingface.co/aurora-m.

Problem

• Multilingual Support
• Memory Retention in Continual Learning
• High Computational Cost

Innovation

• Multilingual Pre-training
• Continuous Learning
• AI Safety Compliance