Simplified Longitudinal Retrieval Experiments: A Case Study on Query Expansion and Document Boosting

2025-09-22

AI Summary

Traditional information retrieval evaluation—e.g., the Cranfield paradigm—ignores temporal dynamics, making it difficult to model the co-evolution of query intent and document content. Consequently, longitudinal evaluation requires labor-intensive, ad-hoc experimental logic, undermining reproducibility and scalability. This paper proposes a lightweight, declarative framework for longitudinal IR evaluation that unifies time-aware query expansion, document augmentation, and feedback modeling into configurable data interfaces. Built upon *ir_datasets*, it introduces a temporally aware data loader enabling declarative, timestamp-based access to queries, documents, and associated relevance judgments. The framework drastically reduces experimental code complexity while preserving evaluation fidelity. It successfully reproduced the LongEval 2024 benchmark, empirically demonstrating significant improvements in reproducibility, scalability, and development efficiency—without compromising assessment accuracy.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Natural Language Processing: Information Extraction
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

The longitudinal evaluation of retrieval systems aims to capture how information needs and documents evolve over time. However, classical Cranfield-style retrieval evaluations only consist of a static set of queries and documents and thereby miss time as an evaluation dimension. Therefore, longitudinal evaluations need to complement retrieval toolkits with custom logic. This custom logic increases the complexity of research software, which might reduce the reproducibility and extensibility of experiments. Based on our submissions to the 2024 edition of LongEval, we propose a custom extension of ir_datasets for longitudinal retrieval experiments. This extension allows for declaratively, instead of imperatively, describing important aspects of longitudinal retrieval experiments, e.g., which queries, documents, and/or relevance feedback are available at which point in time. We reimplement our submissions to LongEval 2024 against our new ir_datasets extension, and find that the declarative access can reduce the complexity of the code.

Problem

• Classical retrieval evaluations lack time dimension for evolving information needs
• Custom logic for longitudinal experiments increases software complexity and reduces reproducibility
• Need declarative approach to describe temporal aspects of queries and documents

Innovation

• Custom ir_datasets extension for longitudinal retrieval
• Declarative description of queries and documents over time
• Reduces code complexity for time-based evaluation experiments