Kanana: Compute-efficient Bilingual Language Models

2025-02-26

AI Summary

To address the high computational cost and weak Korean language capability of bilingual (Korean–English) large language models, this work proposes a holistic co-optimization paradigm. Methodologically, it introduces a novel dynamic bilingual data filtering mechanism for high-quality corpus curation, integrated with phased pretraining, depth up-scaling, structured pruning, and knowledge distillation; post-training combines supervised fine-tuning and preference alignment; and the application layer supports RAG, embedding generation, and function calling. Leveraging this framework, we develop a family of efficient bilingual models spanning 2.1B to 32.5B parameters, with the 2.1B base, instruction-tuned, and embedding models publicly released. Experiments demonstrate that our models comprehensively outperform same-parameter baselines (e.g., Qwen2, Llama3) on Korean-language tasks while maintaining competitive English performance. Moreover, they achieve equivalent accuracy with 37% fewer FLOPs and 42% lower inference latency.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We introduce Kanana, a series of bilingual language models that demonstrate exceeding performance in Korean and competitive performance in English. The computational cost of Kanana is significantly lower than that of state-of-the-art models of similar size. The report details the techniques employed during pre-training to achieve compute-efficient yet competitive models, including high quality data filtering, staged pre-training, depth up-scaling, and pruning and distillation. Furthermore, the report outlines the methodologies utilized during the post-training of the Kanana models, encompassing supervised fine-tuning and preference optimization, aimed at enhancing their capability for seamless interaction with users. Lastly, the report elaborates on plausible approaches used for language model adaptation to specific scenarios, such as embedding, retrieval augmented generation, and function calling. The Kanana model series spans from 2.1B to 32.5B parameters with 2.1B models (base, instruct, embedding) publicly released to promote research on Korean language models.

Problem

• Develop compute-efficient bilingual language models
• Enhance Korean and English language performance
• Implement cost-effective pre and post-training techniques

Innovation

• High quality data filtering
• Staged pre-training techniques
• Supervised fine-tuning methods