Entity Labels Are Not Entity Signals: A Framework for Observable Relevance in Document Re-Ranking

2026-06-14

AI Summary

This work addresses a critical flaw in existing entity-aware retrieval methods, which erroneously treat Conceptual Entity Relevance (CER) as a reliable ranking signal while neglecting the observational reliability of entity linking outputs. The study formally distinguishes CER from Observable Entity Relevance (OER) and proposes leveraging OER—derived from entities demonstrably present and discriminative in actual documents—as the supervision signal for document re-ranking. An OER-driven supervised learning framework is established through entity linking analysis, human annotation, and κ-agreement validation. Experimental results demonstrate that the proposed approach improves non-relevant document pruning efficiency by up to tenfold and achieves a statistically significant gain of 0.051 in Mean Average Precision (MAP) over BM25 in open-domain retrieval settings.

Abstract

Entity-aware document retrieval uses query-associated entities as ranking signals, assuming that semantically relevant entities are also useful retrieval signals. We show this assumption is insufficient- and explain why. Unlike terms, which are ground-truth observations, entity links are hypotheses produced by an imperfect linker: an entity can be topically central yet provide no discriminative signal if the linker fires indiscriminately across relevant and non-relevant documents. We formalize this as a distinction between Conceptual Entity Relevance (CER)- whether an entity is topically related to a query- and Observable Entity Relevance (OER)- whether its observed presence in a collection discriminates relevant from non-relevant documents. Across four collections and annotation sources including human entity judgments, CER and OER exhibit near-chance agreement ($κ\approx 0$), while OER operationalizations agree substantially ($κ\approx 0.5$), confirming CER as the systematic outlier. CER-based supervision selects topically plausible but weakly discriminative entities, pruning fewer than 4% of non-relevant documents on some collections. Aligning supervision with OER improves non-relevant pruning by up to 10x and open-world MAP by 0.051 over BM25. Our findings motivate a shift from conceptual to observable notions of entity relevance in entity-aware retrieval.

Problem

• entity relevance
• document re-ranking
• entity linking
• retrieval signals
• observable relevance

Innovation

• Observable Entity Relevance
• Entity-aware Retrieval
• Document Re-ranking
• Entity Linking
• Discriminative Signal