iCBIR-Sli: Interpretable Content-Based Image Retrieval with 2D Slice Embeddings

2025-01-03

AI Summary

Existing brain MRI image retrieval methods rely on textual descriptions or 3D models, suffering from semantic gaps, high data requirements, fragmented depth information, and omission of pathological features. This paper proposes the first interpretable content-based MRI image retrieval system, introducing a novel global 2D-slice embedding aggregation mechanism: local representations are extracted via a 2D slice encoder, and cross-slice attention enables structural-preserving low-dimensional global embedding, accompanied by interpretable heatmaps for lesion localization. Notably, the method achieves classification-level retrieval performance without requiring an external classifier. Evaluated on five public datasets—ADNI2, ADNI3, OASIS3, OASIS4, and AIBL—the system attains a top-1 retrieval macro F1 score of 0.859, demonstrating significant improvement in precise localization of Alzheimer’s disease–associated brain regions.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Representation Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Current methods for searching brain MR images rely on text-based approaches, highlighting a significant need for content-based image retrieval (CBIR) systems. Directly applying 3D brain MR images to machine learning models offers the benefit of effectively learning the brain's structure; however, building the generalized model necessitates a large amount of training data. While models that consider depth direction and utilize continuous 2D slices have demonstrated success in segmentation and classification tasks involving 3D data, concerns remain. Specifically, using general 2D slices may lead to the oversight of pathological features and discontinuities in depth direction information. Furthermore, to the best of the authors' knowledge, there have been no attempts to develop a practical CBIR system that preserves the entire brain's structural information. In this study, we propose an interpretable CBIR method for brain MR images, named iCBIR-Sli (Interpretable CBIR with 2D Slice Embedding), which, for the first time globally, utilizes a series of 2D slices. iCBIR-Sli addresses the challenges associated with using 2D slices by effectively aggregating slice information, thereby achieving low-dimensional representations with high completeness, usability, robustness, and interoperability, which are qualities essential for effective CBIR. In retrieval evaluation experiments utilizing five publicly available brain MR datasets (ADNI2/3, OASIS3/4, AIBL) for Alzheimer's disease and cognitively normal, iCBIR-Sli demonstrated top-1 retrieval performance (macro F1 = 0.859), comparable to existing deep learning models explicitly designed for classification, without the need for an external classifier. Additionally, the method provided high interpretability by clearly identifying the brain regions indicative of the searched-for disease.

Problem

• Brain MRI retrieval
• 3D image training
• Information preservation

Innovation

• iCBIR-Sli
• 3D MRI Image Retrieval
• Disease-Related Brain Region Identification