Fine-Grained Vision-Language Pretraining with Organ-Conditioned Pattern Tokens for CT Understanding

2026-07-15

AI Summary

Existing vision–language pretraining methods for CT imaging and radiology reports predominantly rely on global or coarse-grained alignment, which struggles to capture fine-grained radiological patterns reflecting intra-organ heterogeneity. This work proposes OCP-CT, a novel framework that, for the first time, leverages organ-conditioned radiological patterns as alignment units to achieve fine-grained cross-modal correspondence. Building upon global contrastive learning, OCP-CT introduces an organ-conditioned pattern interface that dynamically generates continuous pattern tokens through a sparse mixture-of-experts (MoE) architecture coupled with learnable slot mechanisms. Structured soft targets derived from clinical semantic similarity are employed to align image and text patterns. On the CT-RATE and RAD-ChestCT benchmarks, the model achieves zero-shot abnormality diagnosis AUROCs of 84.5% and 69.9%, respectively, surpassing prior state-of-the-art methods by 6.7 and 0.8 percentage points.

Abstract

Computed tomography (CT) vision-language pretraining from paired volumes and radiology reports is a scalable yet challenging task. Existing methods commonly adopt global scan-report contrast, which is scalable but obscures heterogeneous organ evidence. Meanwhile, direct organ-level alignment remains coarse, since the same anatomy can exhibit multiple distinct radiological appearances. Therefore, pretraining requires a finer alignment unit: the organ-conditioned radiological pattern. In this work, we propose OCP-CT, an organ-conditioned pattern-token alignment framework for CT vision-language pretraining. Specifically, OCP-CT preserves a stable global CT-report contrastive branch and introduces an organ pattern interface: sparse Mixture-of-Experts (MoE) routes image and text tokens according to latent radiological patterns, learnable slots query the routed tokens into continuous pattern tokens, and paired token contrast aligns image-text pattern tokens with structured soft targets built from report-derived clinical similarity. On the publicly available CT-RATE and RAD-ChestCT benchmarks, OCP-CT achieves average AUROCs of 84.5% and 69.9% for zero-shot abnormality diagnosis, respectively. Compared with the strongest prior reported results, these results yield absolute AUROC gains of 6.7 and 0.8 percentage points.

Problem

• fine-grained alignment
• organ-conditioned pattern
• CT vision-language pretraining
• radiological appearance
• heterogeneous organ evidence

Innovation

• organ-conditioned pattern tokens
• fine-grained vision-language pretraining
• Mixture-of-Experts (MoE)
• structured soft targets
• CT understanding