Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion

2024-12-16

arXiv.org

AI Summary

Arabic large language models (LLMs) suffer from severe out-of-vocabulary (OOV) issues and knowledge degradation due to static vocabulary constraints. To address this, we propose a cognitively inspired, progressive vocabulary expansion method—motivated by second-language acquisition—that dynamically incorporates Arabic subwords during pretraining, balancing decoding efficiency and knowledge retention. Our approach integrates an enhanced dynamic Byte-Pair Encoding (BPE) tokenizer, a controllable vocabulary growth strategy, and an Arabic-specific pretraining–evaluation framework. The resulting model, AraLLaMA, achieves state-of-the-art performance across multiple Arabic benchmarks, matching or exceeding the best open-source Arabic LLMs. We fully open-source the model weights, training data, and code. This work is the first to incorporate cognitive linguistic principles into vocabulary learning for LLMs, establishing a scalable, principled paradigm for training LLMs in low-resource languages.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

This paper addresses the critical need for democratizing large language models (LLM) in the Arab world, a region that has seen slower progress in developing models comparable to state-of-the-art offerings like GPT-4 or ChatGPT 3.5, due to a predominant focus on mainstream languages (e.g., English and Chinese). One practical objective for an Arabic LLM is to utilize an Arabic-specific vocabulary for the tokenizer that could speed up decoding. However, using a different vocabulary often leads to a degradation of learned knowledge since many words are initially out-of-vocabulary (OOV) when training starts. Inspired by the vocabulary learning during Second Language (Arabic) Acquisition for humans, the released AraLLaMA employs progressive vocabulary expansion, which is implemented by a modified BPE algorithm that progressively extends the Arabic subwords in its dynamic vocabulary during training, thereby balancing the OOV ratio at every stage. The ablation study demonstrated the effectiveness of Progressive Vocabulary Expansion. Moreover, AraLLaMA achieves decent performance comparable to the best Arabic LLMs across a variety of Arabic benchmarks. Models, training data, benchmarks, and codes will be all open-sourced.

Problem

• Democratizing Arabic large language models to address regional development gaps
• Reducing vocabulary degradation when transitioning to Arabic-specific tokenizers
• Balancing out-of-vocabulary ratios during Arabic language model training

Innovation

• Progressive vocabulary expansion for Arabic LLMs
• Modified BPE algorithm with dynamic vocabulary
• Balancing OOV ratio during training stages