In-Domain Supervised Pathology Report Classification: A Reproducible Pipeline from Data Curation to Production-Matched Evaluation

2026-06-14

AI Summary

This study addresses the performance degradation of pathology report classification models when deployed across different cancer registries due to distributional shift. The authors propose a reproducible, end-to-end supervised classification framework that constructs training and evaluation data closely mirroring real-world deployment scenarios. This is achieved through institution-stratified sampling, separate processing of registry-linked case reports, and blinded estimation of positive rates and label noise. By integrating supervised text classification with threshold optimization constrained by a target false negative rate (FNR), the model achieves substantially improved generalization. Evaluated on a test set of 418,000 reports, the Kentucky model attains an FNR of 0.003, an FPR of 0.097, and an F1 score of 0.922—significantly outperforming baseline approaches, which achieved an F1 score of 0.860.

Abstract

We introduce an in-domain supervised pipeline designed to counter the out-of-distribution performance drop that hampers supervised biomedical NLP models, a problem observed when models trained on pathology reports are moved across cancer registries. Our contribution is a reproducible recipe for training a supervised classifier from routinely collected cancer registry data. It describes how to build the in-domain training set and a production-matched holdout, and to choose operating points that keep the false-negative rate (FNR) very low while keeping reviewer workload manageable. The pipeline standardizes data curation with facility-stratified sampling and separate handling of reports linked to registry cases, and includes a blinded manual audit to estimate positive-case prevalence and label noise. On a 418k-report holdout set, the Kentucky model achieved FNR 0.003 and false-positive rate (FPR) 0.097, improving over the Seattle-trained MOSSAIC OncoID baseline (FNR 0.010, FPR 0.183) and raising F1 from 0.860 to 0.922. In a blinded manual review of 600 reports, estimated positive prevalence declined from 0.500 to 0.398, indicating substantial label noise with errors concentrated in rare primary sites.

Problem

• out-of-distribution
• pathology report classification
• biomedical NLP
• cancer registry
• label noise

Innovation

• in-domain supervision
• production-matched evaluation
• facility-stratified sampling
• label noise estimation
• false-negative rate control