Recall Them All: Retrieval-Augmented Language Models for Long Object List Extraction from Long Documents

2024-05-04

arXiv.org

AI Summary

Existing methods for extracting long object lists—those bearing specific relations to a given subject in lengthy documents—suffer from low recall due to overreliance on local context, failing to capture relevant entities dispersed across paragraphs. Method: We propose L3X, a two-stage framework: (1) retrieval-augmented large language model (LLM) generation to produce high-recall candidate lists; and (2) structured validation and pruning to enhance precision. Contribution/Results: L3X is the first approach to systematically balance the recall–precision trade-off in long-list extraction, integrating retrieval-augmented generation (RAG), prompt engineering, and interpretable verification. Evaluated on multiple long-document benchmarks, L3X achieves over 40% higher recall than baseline LLM-only methods while maintaining high precision—demonstrating substantial gains in both effectiveness and robustness.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Methods for relation extraction from text mostly focus on high precision, at the cost of limited recall. High recall is crucial, though, to populate long lists of object entities that stand in a specific relation with a given subject. Cues for relevant objects can be spread across many passages in long texts. This poses the challenge of extracting long lists from long texts. We present the L3X method which tackles the problem in two stages: (1) recall-oriented generation using a large language model (LLM) with judicious techniques for retrieval augmentation, and (2) precision-oriented scrutinization to validate or prune candidates. Our L3X method outperforms LLM-only generations by a substantial margin.

Problem

• Extract long object lists from long documents
• Improve recall in relation extraction tasks
• Validate and prune candidates for precision

Innovation

• Two-stage L3X method for long list extraction
• Recall-oriented generation with retrieval-augmented LLM
• Precision-oriented scrutinization for candidate validation