Human-Annotated NER Dataset for the Kyrgyz Language

2025-09-23

AI Summary

To address the scarcity of high-quality manually annotated data for Kyrgyz named entity recognition (NER), this study introduces KyrgyzNER—the first large-scale, fine-grained, human-annotated NER dataset for Kyrgyz, comprising 1,499 news articles, 10,900 sentences, and 39,075 entity mentions across 27 entity types. We establish linguistically informed annotation guidelines tailored to Kyrgyz morphology and syntax. We systematically evaluate both traditional conditional random fields (CRF) and multilingual pre-trained language models (e.g., mRoBERTa) on this benchmark. Experimental results demonstrate that mRoBERTa achieves the best trade-off between precision and recall, significantly outperforming CRF-based approaches; other multilingual models also exhibit robust performance. KyrgyzNER fills a critical gap in NER resources for low-resource Turkic languages and serves as a foundational benchmark for cross-lingual information extraction research, providing both empirical evidence and standardized evaluation infrastructure.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Knowledge Representation Languages
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We introduce KyrgyzNER, the first manually annotated named entity recognition dataset for the Kyrgyz language. Comprising 1,499 news articles from the 24.KG news portal, the dataset contains 10,900 sentences and 39,075 entity mentions across 27 named entity classes. We show our annotation scheme, discuss the challenges encountered in the annotation process, and present the descriptive statistics. We also evaluate several named entity recognition models, including traditional sequence labeling approaches based on conditional random fields and state-of-the-art multilingual transformer-based models fine-tuned on our dataset. While all models show difficulties with rare entity categories, models such as the multilingual RoBERTa variant pretrained on a large corpus across many languages achieve a promising balance between precision and recall. These findings emphasize both the challenges and opportunities of using multilingual pretrained models for processing languages with limited resources. Although the multilingual RoBERTa model performed best, other multilingual models yielded comparable results. This suggests that future work exploring more granular annotation schemes may offer deeper insights for Kyrgyz language processing pipelines evaluation.

Problem

• Creating first manually annotated NER dataset for Kyrgyz language
• Evaluating NER models for rare entity categories in Kyrgyz
• Exploring multilingual models for low-resource Kyrgyz language processing

Innovation

• First manually annotated Kyrgyz NER dataset
• Evaluated CRF and multilingual transformer models
• Multilingual RoBERTa achieved best performance balance