EviSearch: A Human in the Loop System for Extracting and Auditing Clinical Evidence for Systematic Reviews

2026-03-23

AI Summary

This work addresses the inefficiency and lack of fine-grained provenance in manually extracting structured evidence from clinical trial PDFs for systematic reviews. The authors propose a multi-agent system that automatically extracts multimodal content—spanning text, tables, and figures—from raw clinical trial PDFs to generate ontology-aligned evidence tables, with verifiable source attribution for every cell. The system integrates a layout-preserving PDF querying agent, a retrieval-guided search agent, and a conflict reconciliation module, augmented by a human-in-the-loop mechanism that triggers page-level expert review upon agent disagreement and logs all annotations and edits to produce iterative supervisory signals. Evaluated on an oncology trial benchmark, the approach significantly outperforms strong baselines, achieving high extraction accuracy, comprehensive provenance coverage, and a substantial reduction in manual curation effort.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Multiagent Systems: Other Foundations of Multi Agent Systems
• Natural Language Processing: Information Extraction

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

We present EviSearch, a multi-agent extraction system that automates the creation of ontology-aligned clinical evidence tables directly from native trial PDFs while guaranteeing per-cell provenance for audit and human verification. EviSearch pairs a PDF-query agent (which preserves rendered layout and figures) with a retrieval-guided search agent and a reconciliation module that forces page-level verification when agents disagree. The pipeline is designed for high-precision extraction across multimodal evidence sources (text, tables, figures) and for generating reviewer-actionable provenance that clinicians can inspect and correct. On a clinician-curated benchmark of oncology trial papers, EviSearch substantially improves extraction accuracy relative to strong parsed-text baselines while providing comprehensive attribution coverage. By logging reconciler decisions and reviewer edits, the system produces structured preference and supervision signals that bootstrap iterative model improvement. EviSearch is intended to accelerate living systematic review workflows, reduce manual curation burden, and provide a safe, auditable path for integrating LLM-based extraction into evidence synthesis pipelines.

Problem

• clinical evidence extraction
• systematic reviews
• human-in-the-loop
• auditability
• PDF document processing

Innovation

• multi-agent extraction
• ontology-aligned evidence tables
• per-cell provenance
• human-in-the-loop auditing
• multimodal clinical evidence