Measuring Hypothesis Testing Errors in the Evaluation of Retrieval Systems

2025-07-10

AI Summary

This study addresses the quantification of discriminative power in query-document relevance judgments (qrels) for information retrieval (IR) evaluation, with particular emphasis on the historically underexamined Type II error (false negatives) and its joint analysis with Type I error (false positives). Method: We systematically introduce Type II error modeling into IR evaluation for the first time and propose replacing conventional significance testing with balanced classification metrics—such as balanced accuracy—as a more principled basis for assessing qrels’ discriminative capability. A unified, comparable measurement framework is thereby established. Results: Empirical hypothesis testing and statistical significance analysis across multiple qrels generation strategies demonstrate that jointly evaluating both error types exposes latent quality deficiencies in qrels more comprehensively than traditional approaches. Balanced classification metrics robustly aggregate discriminative performance, substantially enhancing the reliability and interpretability of IR system evaluation.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

The evaluation of Information Retrieval (IR) systems typically uses query-document pairs with corresponding human-labelled relevance assessments (qrels). These qrels are used to determine if one system is better than another based on average retrieval performance. Acquiring large volumes of human relevance assessments is expensive. Therefore, more efficient relevance assessment approaches have been proposed, necessitating comparisons between qrels to ascertain their efficacy. Discriminative power, i.e. the ability to correctly identify significant differences between systems, is important for drawing accurate conclusions on the robustness of qrels. Previous work has measured the proportion of pairs of systems that are identified as significantly different and has quantified Type I statistical errors. Type I errors lead to incorrect conclusions due to false positive significance tests. We argue that also identifying Type II errors (false negatives) is important as they lead science in the wrong direction. We quantify Type II errors and propose that balanced classification metrics, such as balanced accuracy, can be used to portray the discriminative power of qrels. We perform experiments using qrels generated using alternative relevance assessment methods to investigate measuring hypothesis testing errors in IR evaluation. We find that additional insights into the discriminative power of qrels can be gained by quantifying Type II errors, and that balanced classification metrics can be used to give an overall summary of discriminative power in one, easily comparable, number.

Problem

• Quantifying Type II errors in IR system evaluations
• Assessing discriminative power of relevance assessments (qrels)
• Using balanced metrics to summarize qrels' effectiveness

Innovation

• Quantify Type II errors in IR evaluation
• Use balanced accuracy for discriminative power
• Compare qrels from alternative assessment methods