A Practice of Post-Training on Llama-3 70B with Optimal Selection of Additional Language Mixture Ratio

2024-09-10

arXiv.org

AI Summary

Large language models (LLMs) suffer from constrained Chinese capability gains during continual pretraining (CPT) due to misaligned coupling between the additional language mixture ratio (ALMR) and learning rate (LR). Method: Building upon Llama-3 8B and 70B base models, we systematically investigate the ALMR–LR coupling mechanism. We propose and empirically validate a novel ALMR–LR coupling scaling law, enabling principled hyperparameter transfer across model scales (8B → 70B). Our approach integrates CPT, coordinated hyperparameter search, and domain-adaptive fine-tuning. Results: The method significantly enhances Chinese understanding and generation, while concurrently improving performance on downstream subtasks—including mathematical reasoning, code generation, and sentiment intelligence. The optimized 70B model has been successfully deployed in a production dialogue system, demonstrating industrial-grade robustness and efficacy. Crucially, the scaling law bridges the performance gap between small-scale experimentation and full-scale deployment, offering generalizable guidance for multilingual LLM adaptation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLM) often needs to be Continual Pre-Trained (CPT) to obtain the unfamiliar language skill or adapt into new domains. The huge training cost of CPT often asks for cautious choice of key hyper-parameters such as the mixture ratio of extra language or domain corpus. However, there is no systematic study which bridge the gap between the optimal mixture ratio and the actual model performance, and the gap between experimental scaling law and the actual deployment in the full model size. In this paper, we perform CPT on Llama-3 8B and 70B to enhance its Chinese ability. We study the optimal correlation between the Additional Language Mixture Ratio (ALMR) and the Learning Rate (LR) on the 8B size which directly indicate the optimal experimental set up. By thorough choice of hyper-parameter, and subsequent fine-tuning, the model capability is improved not only on the Chinese-related benchmark, but also some specific domains including math, coding and emotional intelligence. We deploy the final 70B version of LLM on an real-life chat system which obtain satisfying performance.

Problem

• Determines optimal language mixture ratio for continual pre-training
• Bridges gap between experimental scaling and full model deployment
• Enhances Chinese ability and domain performance in LLMs

Innovation

• Optimal Additional Language Mixture Ratio selection
• Hyper-parameter correlation study on 8B model
• Deployment of enhanced 70B model in chat system