GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture

2025-06-11

AI Summary

To address the lag in developing large language models (LLMs) for Russian—largely due to prohibitive computational costs—this work introduces GigaChat, the first open-source family of Mixture-of-Experts (MoE) LLMs specifically designed for Russian. GigaChat leverages Russian-centric pretraining, supervised instruction fine-tuning, and comprehensive multi-stage evaluation on Russian benchmarks (e.g., RuEval, RACE-Ru, XGLUE), achieving high performance while substantially reducing training and inference overhead. The family comprises multiple base model sizes and corresponding instruction-tuned variants, consistently outperforming multilingual baselines such as mT5 and BLOOM on Russian understanding and generation tasks. To foster adoption, three models are publicly released on Hugging Face, accompanied by production-ready interfaces—including a REST API, Telegram bot, and web application—enabling scalable industrial deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Generative large language models (LLMs) have become crucial for modern NLP research and applications across various languages. However, the development of foundational models specifically tailored to the Russian language has been limited, primarily due to the significant computational resources required. This paper introduces the GigaChat family of Russian LLMs, available in various sizes, including base models and instruction-tuned versions. We provide a detailed report on the model architecture, pre-training process, and experiments to guide design choices. In addition, we evaluate their performance on Russian and English benchmarks and compare GigaChat with multilingual analogs. The paper presents a system demonstration of the top-performing models accessible via an API, a Telegram bot, and a Web interface. Furthermore, we have released three open GigaChat models in open-source (https://huggingface.co/ai-sage), aiming to expand NLP research opportunities and support the development of industrial solutions for the Russian language.

Problem

• Develop efficient Russian language models with limited resources
• Compare GigaChat performance with multilingual models
• Expand NLP research opportunities for Russian language

Innovation

• Mixture of Experts for Russian language efficiency
• Open-source GigaChat models on Hugging Face
• API, Telegram bot, and Web interface integration