Multi-Plane Vision Transformer for Hemorrhage Classification Using Axial and Sagittal MRI Data

2025-05-12

AI Summary

Conventional brain hemorrhage detection in multi-planar (axial/sagittal) MRI suffers from information loss due to resampling to a unified plane. Method: We propose a dual-plane Vision Transformer architecture that eliminates resampling by employing a 3D multi-planar ViT backbone, dual-stream Transformer encoders, and cross-plane cross-attention to jointly model heterogeneous view-specific features; modality-indicative embedding vectors explicitly encode anatomical orientation priors. Results: Evaluated on a clinical dataset of over 10,000 cases, our model achieves an AUC of 0.942—outperforming standard ViT by 5.5% and CNN baselines by 1.8%—demonstrating significantly improved robustness to orientation variability and contrast heterogeneity. This work pioneers the integration of cross-view cross-attention and modality-aware embeddings for multi-planar MRI classification, establishing a novel paradigm for resampling-free multi-planar medical image analysis.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Identifying brain hemorrhages from magnetic resonance imaging (MRI) is a critical task for healthcare professionals. The diverse nature of MRI acquisitions with varying contrasts and orientation introduce complexity in identifying hemorrhage using neural networks. For acquisitions with varying orientations, traditional methods often involve resampling images to a fixed plane, which can lead to information loss. To address this, we propose a 3D multi-plane vision transformer (MP-ViT) for hemorrhage classification with varying orientation data. It employs two separate transformer encoders for axial and sagittal contrasts, using cross-attention to integrate information across orientations. MP-ViT also includes a modality indication vector to provide missing contrast information to the model. The effectiveness of the proposed model is demonstrated with extensive experiments on real world clinical dataset consists of 10,084 training, 1,289 validation and 1,496 test subjects. MP-ViT achieved substantial improvement in area under the curve (AUC), outperforming the vision transformer (ViT) by 5.5% and CNN-based architectures by 1.8%. These results highlight the potential of MP-ViT in improving performance for hemorrhage detection when different orientation contrasts are needed.

Problem

• Classifying brain hemorrhages from MRI with varying orientations
• Reducing information loss in traditional resampling methods
• Improving hemorrhage detection using multi-plane vision transformers

Innovation

• Multi-plane ViT for hemorrhage classification
• Cross-attention integrates axial and sagittal data
• Modality vector provides missing contrast information