Vikhr: Constructing a State-of-the-art Bilingual Open-Source Instruction-Following Large Language Model for Russian

2024-05-22

MRL

AI Summary

To address the poor generation quality and low computational efficiency of existing large language models for Russian, this paper introduces Vikhr—the first high-performance, bilingual, open-source instruction-following model natively optimized for Russian. Methodologically, Vikhr employs full-parameter continual pretraining followed by supervised instruction fine-tuning, deliberately avoiding parameter-efficient adaptations such as LoRA to achieve vocabulary-level native Russian support. Built upon the Mistral architecture, it features a custom Russian–English tokenizer, alongside substantial expansion of high-quality Russian instruction data and pretraining corpora, enhanced by multi-stage data cleaning and synthetic data generation. Experiments demonstrate that Vikhr establishes new state-of-the-art results among open-source models on multiple Russian-language benchmarks, with several metrics surpassing those of proprietary commercial models. All model weights, datasets, and training code are publicly released.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

There has been a surge in the development of various Large Language Models (LLMs). However, text generation for languages other than English often faces significant challenges, including poor generation quality and reduced computational performance due to the disproportionate representation of tokens in the model’s vocabulary. In this work, we address these issues by developing a pipeline for adaptation of English-oriented pre-trained models to other languages and constructing efficient bilingual LLMs. Using this pipeline, we construct Vikhr, a state-of-the-art bilingual open-source instruction-following LLM designed specifically for the Russian language. “Vikhr” refers to the name of the Mistral LLM series and means a “strong gust of wind.”Unlike previous Russian-language models that typically rely on LoRA adapters on top of English-oriented models, sacrificing performance for lower training costs, Vikhr features an adapted tokenizer vocabulary and undergoes the continued pre-training and instruction tuning of all weights. This not only enhances the model’s performance but also significantly improves its computational and contextual efficiency.The remarkable performance of Vikhr across various Russian-language benchmarks can also be attributed to our efforts in expanding instruction datasets and corpora for continued pre-training. Vikhr not only sets the new state of the art among open-source LLMs for Russian but even outperforms some proprietary closed-source models on certain benchmarks. The model weights, instruction sets, and code are publicly available.

Problem

• Develop bilingual LLMs for Russian language
• Adapt English-oriented models to other languages
• Enhance computational and contextual efficiency

Innovation

• Adapts English models to Russian
• Enhanced tokenizer and pre-training
• Expanded Russian instruction datasets