brat: Aligned Multi-View Embeddings for Brain MRI Analysis

2025-12-21

AI Summary

Analyzing subtle, spatially sparse, and viewpoint-dependent lesions in brain MRI remains challenging. Method: We introduce the largest publicly available paired MRI–clinical report dataset to date (80K samples, a 10× scale-up), and propose a multi-view alignment representation learning framework featuring: (i) a novel implicit query-feature matching mechanism; (ii) quality- and diversity-driven multi-view embedding alignment; and (iii) integration of 3D slice-level feature disentanglement/aggregation with a document-retrieval-inspired cross-modal pretraining paradigm. Contribution/Results: Our approach achieves state-of-the-art performance across both vision-language understanding and pure-vision medical tasks. We open-source the BRAT foundation model, enabling zero-shot transfer and clinical report generation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present brat (brain report alignment transformer), a multi-view representation learning framework for brain magnetic resonance imaging (MRI) trained on MRIs paired with clinical reports. Brain MRIs present unique challenges due to the presence of numerous, highly varied, and often subtle abnormalities that are localized to a few slices within a 3D volume. To address these challenges, we introduce a brain MRI dataset $10 imes$ larger than existing ones, containing approximately 80,000 3D scans with corresponding radiology reports, and propose a multi-view pre-training approach inspired by advances in document retrieval. We develop an implicit query-feature matching mechanism and adopt concepts from quality-diversity to obtain multi-view embeddings of MRIs that are aligned with the clinical features given by report sentences. We evaluate our approach across multiple vision-language and vision tasks, demonstrating substantial performance improvements. The brat foundation models are publicly released.

Problem

• Develops a multi-view framework for brain MRI analysis
• Addresses challenges of varied abnormalities in 3D MRI scans
• Aligns MRI embeddings with clinical report features

Innovation

• Multi-view representation learning for brain MRI analysis
• Implicit query-feature matching with clinical report alignment
• Quality-diversity inspired embeddings for localized abnormality detection