Qorgau: Evaluating LLM Safety in Kazakh-Russian Bilingual Contexts

2025-02-19

AI Summary

This work addresses safety risks arising from harmful content generation by large language models (LLMs) in the Kazakh–Russian bilingual context—a critical yet underexplored setting where low-resource Kazakh intersects with high-resource Russian. To bridge this gap, we introduce Qorgau, the first bilingual safety evaluation dataset tailored to Kazakhstan’s multilingual society, featuring localized risk taxonomies, bilingual adversarial prompt engineering, and rigorous human validation. Our systematic analysis reveals, for the first time, significant cross-lingual safety disparities: mainstream LLMs exhibit substantially weaker safety performance in Kazakh compared to Russian. This finding underscores the inadequacy of monolingual or English-centric safety benchmarks for multilingual deployment. We thus establish a region-specific, linguistically collaborative safety evaluation paradigm and empirically demonstrate that bilingual, locally grounded evaluation datasets are both necessary and effective for enhancing LLMs’ localized safety capabilities.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Security and Privacy: Large-scale security measurements

Abstract

Large language models (LLMs) are known to have the potential to generate harmful content, posing risks to users. While significant progress has been made in developing taxonomies for LLM risks and safety evaluation prompts, most studies have focused on monolingual contexts, primarily in English. However, language- and region-specific risks in bilingual contexts are often overlooked, and core findings can diverge from those in monolingual settings. In this paper, we introduce Qorgau, a novel dataset specifically designed for safety evaluation in Kazakh and Russian, reflecting the unique bilingual context in Kazakhstan, where both Kazakh (a low-resource language) and Russian (a high-resource language) are spoken. Experiments with both multilingual and language-specific LLMs reveal notable differences in safety performance, emphasizing the need for tailored, region-specific datasets to ensure the responsible and safe deployment of LLMs in countries like Kazakhstan. Warning: this paper contains example data that may be offensive, harmful, or biased.

Problem

• Assessing LLM safety in bilingual contexts
• Focusing on Kazakh-Russian language pair
• Developing region-specific safety evaluation datasets

Innovation

• Bilingual safety evaluation dataset
• Kazakh-Russian specific LLM testing
• Region-tailored multilingual model assessment