Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning

2026-06-23

AI Summary

This work addresses the limitation of conventional CLIP-style pretraining in aligning 3D medical images with structured radiology reports, which relies solely on global features and overlooks anatomical region-level details. The authors propose ConQuer, a novel approach that extends global alignment with concept-level local alignment: radiology reports are partitioned by anatomical regions, and unsupervised cross-attention queries adaptively aggregate corresponding image features for each concept, followed by independent contrastive learning per concept. This method achieves, for the first time, concept-level vision–language alignment without requiring segmentation masks or spatial supervision, with each query automatically generating attention maps focused on specific anatomical regions—enhancing both performance and interpretability. The resulting Jolia model, trained under this framework, substantially outperforms CLIP baselines on thoracic and abdominal CT scans, establishing new state-of-the-art results across multiple public benchmarks in lesion classification, report generation, and cross-institutional transfer tasks.

Abstract

Vision-language contrastive pretraining has become the dominant recipe for 3D medical foundation models, leveraging the large volumes of paired scans and reports produced in clinical practice. However, medical images usually span dozens of organs, and radiological reports are much longer than typical natural image captions and are composed of multiple structured sections. CLIP-style pretraining compresses this structure by encoding each modality into a single global token, at the risk of losing important details. We introduce ConQuer (Concept Queries), an image-text pretraining method that augments CLIP's global alignment with a set of localized alignments, one per concept. ConQuer splits the report into concept-specific sections and learns cross-attention queries that pool the matching image features without using any segmentation mask or spatial supervision. Contrastive learning is then applied independently for each concept. Concepts can be any unit of semantic localization; here, they are anatomical regions, one query per organ or gross body region. As a byproduct, each query learns attention maps focused on its concept, providing built-in spatial interpretability. We use ConQuer to train Jolia, a 3D CT foundation model on chest and abdominal CT. Jolia consistently outperforms a CLIP baseline on findings classification, report generation, and cross-center transfer, and sets a new state of the art across multiple public benchmarks. Jolia's weights will be released upon acceptance.

Problem

• vision-language alignment
• 3D medical imaging
• contrastive learning
• anatomical localization
• radiology reports

Innovation

• concept-level alignment
• vision-language pretraining
• 3D medical foundation model
• cross-attention queries
• spatial interpretability