Prompt, Plan, Extract: Zero-Shot Agentic LLMs Workflows for Lung Pathology Extraction from Clinical Narratives

2026-06-18

AI Summary

This study addresses the critical need for automated extraction of key clinical information—such as cancer staging—from lung pathology reports to support tumor registry efforts. Traditional approaches rely heavily on costly annotated data and are prone to cascading errors due to pipeline fragmentation. To overcome these limitations, this work proposes the first zero-shot agent-based workflow specifically designed for lung pathology information extraction. Without any task-specific training, the framework leverages five open-source large language models in an end-to-end manner, integrating prompt engineering and planning mechanisms to extract 13 standardized pathology fields. Evaluated against tumor registry benchmarks, the best-performing model, GPT-OSS-20B, achieves a Micro-F1 score of 0.893 (recall: 0.949) under zero-shot settings—approaching the performance of the supervised baseline GatorTron (0.960)—thereby substantially reducing dependence on labeled data and eliminating cascading error propagation.

Abstract

Information extraction from pathology reports is essential for cancer staging, tumor registry population. Yet key data remains embedded in narrative reports, making manual extraction labor-intensive and error-prone. Traditional supervised Natural Language Processing pipelines address this through fully supervised Named Entity Recognition and Relation Extraction, but require expensive manual annotation and suffer cascading failures when upstream entities are missed. In this study, we developed a zero-shot, agentic workflow, and evaluated five open-source generative Large Language Models (LLMs) to populate 13 College of American Pathologists synoptic fields from lung resection pathology reports. We compared them against a state-of-the-art supervised GatorTron NER-RE baseline using a novel, registry-aligned evaluation framework. The baseline achieved Micro-F1of 0.960, while the best zero-shot model (GPT-OSS-20B) achieved Micro-F1 of 0.893 (recall: 0.949), accurately extracting complex relations like Pathologic Stage without task-specific training. These results suggest that open-source, zero-shot agentic LLMs are a low-cost solution for extracting lung pathology information.

Problem

• information extraction
• lung pathology
• clinical narratives
• zero-shot learning
• cancer staging

Innovation

• zero-shot
• agentic LLMs
• information extraction
• lung pathology
• clinical narratives