Knowledge Elicitation with Large Language Models for Interpretable Cancer Stage Identification from Pathology Reports

2025-11-02

AI Summary

To address the challenges of TNM staging extraction from unstructured pathology reports—namely, heavy reliance on labeled data, limited interpretability, and poor generalizability—this paper proposes two unsupervised knowledge extraction frameworks. KEwLTM employs iterative prompting to enable large language models (LLMs) to autonomously induce domain-specific rules; KEwRAG integrates retrieval-augmented generation (RAG) with pre-extracted clinical guidelines. Both leverage zero-shot chain-of-thought reasoning and are validated on TCGA breast cancer pathology reports without manual annotations. Experiments show that KEwLTM achieves superior performance when LLM reasoning is robust, whereas KEwRAG demonstrates greater resilience when zero-shot chain-of-thought fails. Collectively, the frameworks significantly advance automated TNM staging and enhance clinical interpretability, establishing an efficient, transparent, and transferable paradigm for low-resource medical text understanding.

Technology Category

• Natural Language Processing: Information Extraction
• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Cancer staging is critical for patient prognosis and treatment planning, yet extracting pathologic TNM staging from unstructured pathology reports poses a persistent challenge. Existing natural language processing (NLP) and machine learning (ML) strategies often depend on large annotated datasets, limiting their scalability and adaptability. In this study, we introduce two Knowledge Elicitation methods designed to overcome these limitations by enabling large language models (LLMs) to induce and apply domain-specific rules for cancer staging. The first, Knowledge Elicitation with Long-Term Memory (KEwLTM), uses an iterative prompting strategy to derive staging rules directly from unannotated pathology reports, without requiring ground-truth labels. The second, Knowledge Elicitation with Retrieval-Augmented Generation (KEwRAG), employs a variation of RAG where rules are pre-extracted from relevant guidelines in a single step and then applied, enhancing interpretability and avoiding repeated retrieval overhead. We leverage the ability of LLMs to apply broad knowledge learned during pre-training to new tasks. Using breast cancer pathology reports from the TCGA dataset, we evaluate their performance in identifying T and N stages, comparing them against various baseline approaches on two open-source LLMs. Our results indicate that KEwLTM outperforms KEwRAG when Zero-Shot Chain-of-Thought (ZSCOT) inference is effective, whereas KEwRAG achieves better performance when ZSCOT inference is less effective. Both methods offer transparent, interpretable interfaces by making the induced rules explicit. These findings highlight the promise of our Knowledge Elicitation methods as scalable, high-performing solutions for automated cancer staging with enhanced interpretability, particularly in clinical settings with limited annotated data.

Problem

• Extracting cancer staging from unstructured pathology reports automatically
• Overcoming dependency on large annotated datasets for NLP methods
• Developing interpretable AI solutions for clinical cancer staging

Innovation

• Knowledge Elicitation derives rules from unannotated pathology reports
• KEwLTM uses iterative prompting without ground-truth labels
• KEwRAG pre-extracts rules from guidelines for interpretability