ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral

2026-08-11

AI Summary

This work addresses the challenge of efficiently triaging pulmonary nodules by reducing reliance on specialized deep learning models that require extensive annotated data. The authors propose a three-pronged framework that leverages a general-purpose large language model (LLM) to perform zero-shot inference based on structured radiology reports, incorporates confidence calibration to ensure safety, and selectively refers low-confidence cases to a dedicated model. The approach provides theoretical guarantees on the joint error bound and excess risk, demonstrating that linguistic representations encode diagnostically relevant information. Evaluated on the LIDC-IDRI dataset, the method achieves an F1 score of 88.22% and an AUC of 0.92, with the LLM independently handling 76.5% of cases—substantially diminishing dependence on specialized models.

Abstract

Pulmonary nodule malignancy prediction typically depends on image-trained specialist deep learning (DL) models that require substantial annotated imaging data and task-specific training. We investigate whether a generalist large language model (LLM), reading only a faithful natural-language rendering of standard nodule attributes, can serve as a calibrated triage layer. We propose ConfTriage, a confidence-calibrated method built on three pillars: language as the modality, calibration as the safety mechanism, and a selective specialist DL backstop for low-confidence cases. We prove two guarantees: a finite-sample combined-error bound yielding an explicit per-threshold operational certificate, and an oracle inequality showing that excess risk over the Bayes-optimal deferral classifier is controlled by the L1 calibration error of the LLM probability. A controlled seven-way input ablation across five frontier LLMs on LIDC-IDRI shows that natural-language descriptions dominate the diagnostic signal, while low-level image statistics are essentially diagnostically vacuous. ConfTriage achieved an F1 score of 88.22% and an AUC of 0.92, resolving 76.5% of cases using zero-shot LLM inference alone and referring only uncertain cases to the specialist DL backstop. These results demonstrate that clinically meaningful diagnostic information can be captured through structured radiological descriptions and leveraged by calibrated LLMs for selective referral. The framework suggests a practical pathway for combining generalist LLM prediction with specialist AI models in medical decision-support systems. Source code is publicly available at https://github.com/rabiul-ai/ConfTriage.

Problem

• pulmonary nodule malignancy
• large language model
• triage
• calibration
• specialist deferral

Innovation

• calibrated LLM
• selective deferral
• natural-language triage
• confidence calibration
• medical decision support