CancerGUIDE: Cancer Guideline Understanding via Internal Disagreement Estimation

2025-09-08

AI Summary

Clinical decision-making for non-small cell lung cancer (NSCLC) faces challenges in time-consuming, error-prone, and expert-dependent interpretation of NCCN guidelines. Method: We construct the first longitudinal NSCLC clinical dataset and propose a large language model (LLM)-based agent framework for automated, individualized treatment pathway generation. To ensure regulatory compliance and clinical interpretability while reducing annotation costs, we innovatively integrate human annotation with internal model inconsistency estimation to design a confidence-calibrated meta-classifier. Results: Evaluated on 121 real-world cases, our system achieves high concordance with expert consensus (Spearman r = 0.88, RMSE = 0.08) and attains an AUROC of 0.800 for the meta-classifier, demonstrating robustness, interpretability, and scalability.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The National Comprehensive Cancer Network (NCCN) provides evidence-based guidelines for cancer treatment. Translating complex patient presentations into guideline-compliant treatment recommendations is time-intensive, requires specialized expertise, and is prone to error. Advances in large language model (LLM) capabilities promise to reduce the time required to generate treatment recommendations and improve accuracy. We present an LLM agent-based approach to automatically generate guideline-concordant treatment trajectories for patients with non-small cell lung cancer (NSCLC). Our contributions are threefold. First, we construct a novel longitudinal dataset of 121 cases of NSCLC patients that includes clinical encounters, diagnostic results, and medical histories, each expertly annotated with the corresponding NCCN guideline trajectories by board-certified oncologists. Second, we demonstrate that existing LLMs possess domain-specific knowledge that enables high-quality proxy benchmark generation for both model development and evaluation, achieving strong correlation (Spearman coefficient r=0.88, RMSE = 0.08) with expert-annotated benchmarks. Third, we develop a hybrid approach combining expensive human annotations with model consistency information to create both the agent framework that predicts the relevant guidelines for a patient, as well as a meta-classifier that verifies prediction accuracy with calibrated confidence scores for treatment recommendations (AUROC=0.800), a critical capability for communicating the accuracy of outputs, custom-tailoring tradeoffs in performance, and supporting regulatory compliance. This work establishes a framework for clinically viable LLM-based guideline adherence systems that balance accuracy, interpretability, and regulatory requirements while reducing annotation costs, providing a scalable pathway toward automated clinical decision support.

Problem

• Translating complex patient data into guideline-compliant cancer treatment recommendations
• Reducing time and expertise required for accurate NCCN guideline implementation
• Developing clinically viable LLM systems for automated cancer decision support

Innovation

• LLM agent-based approach for guideline-concordant NSCLC treatment
• Hybrid method combining human annotations with model consistency
• Meta-classifier with calibrated confidence scores for verification