OrdRankBen: A Novel Ranking Benchmark for Ordinal Relevance in NLP

2025-03-02

AI Summary

NLP ranking evaluation has long suffered from coarse-grained relevance labeling: binary labels fail to distinguish degrees of relevance, while continuous scores lack explicit ordinal structure, hindering fine-grained discrimination. To address this, we introduce OrdRankBen—the first NLP ranking benchmark explicitly designed for ordinal relevance. Its core innovation lies in structured ordinal relevance labels (e.g., “strongly relevant > moderately relevant > weakly relevant > irrelevant”) and two real-world datasets capturing diverse ordinal label distributions. We employ a hybrid construction methodology combining human annotation with controlled distribution sampling, enabling unified evaluation of ranking-specific LMs, general-purpose LLMs, and dedicated ranking LLMs. Experiments demonstrate that ordinal modeling significantly enhances model sensitivity to subtle relevance distinctions, yielding more precise and robust ranking performance characterization across diverse model architectures.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Ontologies

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

The evaluation of ranking tasks remains a significant challenge in natural language processing (NLP), particularly due to the lack of direct labels for results in real-world scenarios. Benchmark datasets play a crucial role in providing standardized testbeds that ensure fair comparisons, enhance reproducibility, and enable progress tracking, facilitating rigorous assessment and continuous improvement of ranking models. Existing NLP ranking benchmarks typically use binary relevance labels or continuous relevance scores, neglecting ordinal relevance scores. However, binary labels oversimplify relevance distinctions, while continuous scores lack a clear ordinal structure, making it challenging to capture nuanced ranking differences effectively. To address these challenges, we introduce OrdRankBen, a novel benchmark designed to capture multi-granularity relevance distinctions. Unlike conventional benchmarks, OrdRankBen incorporates structured ordinal labels, enabling more precise ranking evaluations. Given the absence of suitable datasets for ordinal relevance ranking in NLP, we constructed two datasets with distinct ordinal label distributions. We further evaluate various models for three model types, ranking-based language models, general large language models, and ranking-focused large language models on these datasets. Experimental results show that ordinal relevance modeling provides a more precise evaluation of ranking models, improving their ability to distinguish multi-granularity differences among ranked items-crucial for tasks that demand fine-grained relevance differentiation.

Problem

• Addresses lack of ordinal relevance labels in NLP ranking tasks.
• Introduces OrdRankBen for multi-granularity relevance distinctions.
• Evaluates models using structured ordinal labels for precise ranking.

Innovation

• Introduces OrdRankBen with structured ordinal labels
• Constructs datasets for ordinal relevance ranking
• Evaluates models using multi-granularity relevance distinctions