REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources

2025-04-10

AI Summary

Existing retrieval test collections suffer from poor cross-task generalization and high reuse costs. This paper introduces the “test collection resurrection” paradigm: leveraging PDF full-text and table structure parsing, context-aware information extraction, and large language models to generate high-quality synthetic relevance labels—thereby reactivating legacy test sets. The approach supports dual enhancement paths—extraction-based and synthesis-based—and optionally incorporates human-in-the-loop verification to balance efficiency and reliability. Experiments on TREC-COVID demonstrate that incorporating tabular information significantly improves RAG system performance while reducing reliance on manual annotation, validating the method’s effectiveness and scalability under resource-constrained conditions. To our knowledge, this is the first work to systematically address the tension between test collection staticity and task dynamism, establishing a sustainable, low-cost pathway for constructing adaptive retrieval evaluation benchmarks.

Technology Category

• Natural Language Processing: Information Extraction
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Retrieval test collections are essential for evaluating information retrieval systems, yet they often lack generalizability across tasks. To overcome this limitation, we introduce REANIMATOR, a versatile framework designed to enable the repurposing of existing test collections by enriching them with extracted and synthetic resources. REANIMATOR enhances test collections from PDF files by parsing full texts and machine-readable tables, as well as related contextual information. It then employs state-of-the-art large language models to produce synthetic relevance labels. Including an optional human-in-the-loop step can help validate the resources that have been extracted and generated. We demonstrate its potential with a revitalized version of the TREC-COVID test collection, showcasing the development of a retrieval-augmented generation system and evaluating the impact of tables on retrieval-augmented generation. REANIMATOR enables the reuse of test collections for new applications, lowering costs and broadening the utility of legacy resources.

Problem

• Enhancing test collections with extracted and synthetic resources
• Improving generalizability of retrieval test collections across tasks
• Reusing legacy test collections for new applications cost-effectively

Innovation

• Enriches test collections with extracted and synthetic resources
• Parses PDF texts and tables for contextual information
• Uses large language models to generate synthetic labels