Learning Anatomy-Grounded CT Vision-Language Representations with Organ-Hierarchical Report Knowledge

2026-07-12

AI Summary

Existing vision–language pretraining methods for CT imaging overlook the hierarchical anatomical knowledge embedded in radiology reports, resulting in coarse-grained image–text alignment. This work proposes OKA-CT, a novel framework that incorporates organ-level structural knowledge into pretraining for the first time. It begins by parsing radiology reports to construct an organ-centric semantic hierarchy and leverages organ-conditioned supervision to enhance CT visual representations. The framework further introduces organ-specific semantic soft targets to enable structured contrastive learning and employs a lightweight global branch to aggregate three-dimensional lesion evidence. Evaluated on CT-RATE and RAD-ChestCT, OKA-CT achieves zero-shot abnormality diagnosis AUROCs of 84.9 and 72.2, respectively, substantially outperforming current baselines while also demonstrating superior image–text alignment precision and lesion sensitivity.

Abstract

Medical vision-language pretraining (VLP) from paired CT images and radiology reports enables scalable representation learning, but most existing methods align either whole scans with entire reports or local image regions with text fragments. These formulations underuse a key property of radiology reports: findings are organized around anatomical structures, with abnormalities described by organs, disease concepts, locations, and severity-related attributes. We propose OKA-CT, an organ-hierarchical knowledge-augmented framework for CT-report VLP. OKA-CT first converts free-text reports into organ-conditioned knowledge using radiology report parsing and LLM-assisted semantic structuring. The extracted hierarchy is used across two learning stages. Stage~1 injects anatomy-grounded evidence into the CT visual representation through fine-grained organ-conditioned supervision, while Stage~2 uses organ-specific report evidence to guide structured report-CT contrastive learning, where hierarchy-derived semantic soft targets treat non-paired cases with shared organ-level findings as weak semantic positives rather than uniform negatives. A lightweight query-based global branch further aggregates disease-relevant volumetric evidence for whole-scan representation. On CT-RATE and RAD-ChestCT datasets, OKA-CT achieves zero-shot abnormality diagnosis AUROCs of 84.9 and 72.2, outperforming prior CT VLP baselines. Retrieval and patch-occlusion analyses further show improved report-image alignment and stronger sensitivity to disease-associated anatomical regions.

Problem

• medical vision-language pretraining
• anatomy-grounded representation
• radiology report structure
• organ-hierarchical knowledge
• CT-report alignment

Innovation

• organ-hierarchical knowledge
• anatomy-grounded representation
• vision-language pretraining
• structured contrastive learning
• CT report parsing