Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text

2026-04-18

AI Summary

Existing coreference resolution approaches underperform on morphologically rich languages such as Hebrew, as they assume that mention boundaries align with word boundaries—an assumption that fails in languages featuring clitic pronouns and complex morphology. To address this gap, this work introduces KibutzR, the first fine-grained coreference dataset for Modern Hebrew, which supports mention annotations at word, subword, and multiword levels. The authors also propose a tokenization-aware evaluation protocol tailored to Hebrew’s morphological complexity. Experiments reveal that current large language models exhibit substantially degraded performance on untokenized Hebrew text and lag far behind their English counterparts. Notably, smaller encoder-based models significantly outperform mainstream decoder architectures, highlighting the critical influence of linguistic properties on model selection for coreference resolution.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Knowledge Representation and Reasoning: Knowledge Representation Languages
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Coreference Resolution (CR) is a fundamental NLP task critical for long-form tasks as information extraction, summarization, and many business applications. However, CR methods originally designed for English struggle with Morphologically Rich Languages (MRLs), where mention boundaries do not necessarily align with word boundaries, and a single token may consist of multiple anaphors. CR modeling and evaluation protocols standardly assume that, as in English, words and mentions mostly align. However, this assumption breaks down in MRLs, particularly in the context of LLMs' raw-text processing and end-to-end tasks. To assess and address this challenge, we introduce {\em KibutzR}, the first comprehensive CR dataset for Modern Hebrew, an MRL rich with complex words and pronominal clitics. We deliver an annotated dataset that identifies mentions at word, sub-word and multi-word levels, and propose an evaluation protocol that directly addresses word/morpheme boundary discrepancies. Our experiments show that contemporary LLMs perform significantly worse on Hebrew than on English, and that performance degrades on raw unsegmented text. Crucially, we show an inverse performance-trend in Hebrew relative to English, where smaller encoders perform far better than contemporary decoder models, leaving ample space for investigation and improvement. We deliver a new benchmark for Hebrew coreference resolution and a segmentation-aware evaluation protocol to inform future work on other MRLs.

Problem

• Coreference Resolution
• Morphologically Rich Languages
• Word Boundaries
• Hebrew
• Evaluation Protocol

Innovation

• coreference resolution
• morphologically rich languages
• sub-word mention
• evaluation protocol
• Hebrew NLP