IdentifyMe: A Challenging Long-Context Mention Resolution Benchmark

2024-11-12

arXiv.org

AI Summary

Existing evaluations of large language models (LLMs) on long-context coreference resolution are insufficient, lacking rigorous, fine-grained benchmarks for referential understanding in extended texts. Method: We introduce IdentifyMe—the first multiple-choice question (MCQ) benchmark specifically designed for long-text coreference understanding. It employs heuristic filtering to exclude trivial coreferents, integrates diverse referential types (pronouns, nominals), and incorporates nested structural distractors to increase difficulty. The MCQ format enables granular assessment across referent types and systematic analysis of structural confounds. Contribution/Results: Experiments reveal that GPT-4o achieves 81.9% accuracy, while leading open-weight small models lag by 20–30 percentage points. Pronominal coreference proves significantly more challenging than nominal coreference, and models exhibit severe entity confusion within nested mentions. IdentifyMe establishes a more rigorous, decomposable evaluation paradigm for long-context coreference resolution, enabling targeted diagnosis of model capabilities and limitations.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent evaluations of LLMs on coreference resolution have revealed that traditional output formats and evaluation metrics do not fully capture the models' referential understanding. To address this, we introduce IdentifyMe, a new benchmark for mention resolution presented in a multiple-choice question (MCQ) format, commonly used for evaluating LLMs. IdentifyMe features long narratives and employs heuristics to exclude easily identifiable mentions, creating a more challenging task. The benchmark also consists of a curated mixture of different mention types and corresponding entities, allowing for a fine-grained analysis of model performance. We evaluate both closed- and open source LLMs on IdentifyMe and observe a significant performance gap (20-30%) between the state-of-the-art sub-10B open models vs. closed ones. We observe that pronominal mentions, which have limited surface information, are typically much harder for models to resolve than nominal mentions. Additionally, we find that LLMs often confuse entities when their mentions overlap in nested structures. The highest-scoring model, GPT-4o, achieves 81.9% accuracy, highlighting the strong referential capabilities of state-of-the-art LLMs while also indicating room for further improvement.

Problem

• Evaluates LLMs on challenging long-context mention resolution
• Introduces IdentifyMe benchmark with diverse mention types
• Reveals performance gaps in pronominal vs nominal mentions

Innovation

• Introduces MCQ format for mention resolution
• Uses long narratives and heuristic exclusions
• Evaluates mixed mention types for analysis