Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark

2026-04-14

AI Summary

This work addresses the limited language coverage and inconsistent annotation standards in existing named entity recognition (NER) benchmarks, which hinder systematic evaluation of cross-lingual NER capabilities in multilingual large language models. To bridge this gap, the authors propose UNER v2, a large-scale multilingual NER benchmark built upon a unified label schema and linguistically informed guidelines across diverse languages. Leveraging a community-driven, human-centric annotation pipeline, UNER v2 achieves high inter-annotator consistency and scalability. The dataset substantially expands language coverage compared to prior efforts, filling critical gaps in multilingual NER evaluation and establishing a continuously evolving gold standard for cross-lingual entity recognition research.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

While multilingual language models promise to bring the benefits of LLMs to speakers of many languages, gold-standard evaluation benchmarks in most languages to interrogate these assumptions remain scarce. The Universal NER project, now entering its fourth year, is dedicated to building gold-standard multilingual Named Entity Recognition (NER) benchmark datasets. Inspired by existing massively multilingual efforts for other core NLP tasks (e.g., Universal Dependencies), the project uses a general tagset and thorough annotation guidelines to collect standardized, cross-lingual annotations of named entity spans. The first installment (UNER v1) was released in 2024, and the project has continued and expanded since then, with various organizers, annotators, and collaborators in an active community.

Problem

• Named Entity Recognition
• multilingual benchmark
• gold-standard dataset
• cross-lingual evaluation
• low-resource languages

Innovation

• Massively Multilingual
• Named Entity Recognition
• Gold-standard Benchmark
• Cross-lingual Annotation
• Universal Tagset