Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series

2026-04-12

AI Summary

This work addresses the inefficiency of generic large language model tokenizers in handling the complex morphological structure of Polish, which leads to tokenization redundancy, high inference costs, and constrained effective context length. To overcome these limitations, the authors introduce the first Polish-specific tokenizer and integrate it into the Bielik v3 7B and 11B model series through FOCUS embedding initialization, multi-stage pretraining, and a comprehensive alignment pipeline comprising supervised fine-tuning (SFT), direct preference optimization (DPO), and group relative policy optimization (GRPO) with verifiable rewards. This approach substantially reduces token sequence length and computational overhead, extends the effective context window, and achieves significant performance gains across multiple localized benchmarks.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The development of the Bielik v3 PL series, encompassing both the 7B and 11B parameter variants, represents a significant milestone in the field of language-specific large language model (LLM) optimization. While general-purpose models often demonstrate impressive multilingual capabilities, they frequently suffer from a fundamental architectural inefficiency: the use of universal tokenizers. These tokenizers, typically designed to cover a broad spectrum of languages, often fail to capture the morphological nuances of specific languages like Polish, leading to higher fertility ratios, increased inference costs, and restricted effective context windows. This report details the transition from the universal Mistral-based tokenization to a dedicated Polish-optimized vocabulary for the Bielik v3 models, exploring the FOCUS-based embedding initialization, the multi-stage pretraining curriculum, and the subsequent post-training alignment involving Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning through Group Relative Policy Optimization with verifiable rewards.

Problem

• tokenizer optimization
• Polish language modeling
• morphological complexity
• fertility ratio
• context window

Innovation

• tokenizer optimization
• language-specific LLM
• FOCUS embedding initialization
• multi-stage pretraining
• preference optimization