Content-based 3D Image Retrieval and a ColBERT-inspired Re-ranking for Tumor Flagging and Staging

2025-07-23

AI Summary

The exponential growth of medical imaging data hinders radiologists’ ability to efficiently retrieve similar cases, while existing content-based image retrieval (CBIR) systems suffer from a lack of standardized evaluation and clinical integration. To address this, we propose a novel framework for content-based retrieval and re-ranking of three-dimensional (3D) medical images—requiring neither pre-segmentation nor organ-specific annotations—and directly supporting tumor marking and staging tasks. Our core contribution is C-MIR, an adaptation of ColBERT’s late interaction mechanism to 3D medical imaging, enabling end-to-end lesion localization and cross-anatomical-site retrieval. By integrating deep feature extraction with multimodal contextual fusion, the framework achieves statistically significant improvements in retrieval performance across four tumor sites (colon, lung, liver, and breast; *p* < 0.05). It demonstrates strong practical potential for deployment within clinical PACS environments handling large-scale, unstructured medical imaging archives.

Technology Category

• Computer Vision: Image and Video Retrieval
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

The increasing volume of medical images poses challenges for radiologists in retrieving relevant cases. Content-based image retrieval (CBIR) systems offer potential for efficient access to similar cases, yet lack standardized evaluation and comprehensive studies. Building on prior studies for tumor characterization via CBIR, this study advances CBIR research for volumetric medical images through three key contributions: (1) a framework eliminating reliance on pre-segmented data and organ-specific datasets, aligning with large and unstructured image archiving systems, i.e. PACS in clinical practice; (2) introduction of C-MIR, a novel volumetric re-ranking method adapting ColBERT's contextualized late interaction mechanism for 3D medical imaging; (3) comprehensive evaluation across four tumor sites using three feature extractors and three database configurations. Our evaluations highlight the significant advantages of C-MIR. We demonstrate the successful adaptation of the late interaction principle to volumetric medical images, enabling effective context-aware re-ranking. A key finding is C-MIR's ability to effectively localize the region of interest, eliminating the need for pre-segmentation of datasets and offering a computationally efficient alternative to systems relying on expensive data enrichment steps. C-MIR demonstrates promising improvements in tumor flagging, achieving improved performance, particularly for colon and lung tumors (p<0.05). C-MIR also shows potential for improving tumor staging, warranting further exploration of its capabilities. Ultimately, our work seeks to bridge the gap between advanced retrieval techniques and their practical applications in healthcare, paving the way for improved diagnostic processes.

Problem

• Enhancing 3D medical image retrieval for tumor analysis
• Eliminating need for pre-segmented data in CBIR systems
• Improving tumor flagging and staging with C-MIR method

Innovation

• Framework eliminates need for pre-segmented data
• C-MIR adapts ColBERT for 3D medical imaging
• Effective context-aware re-ranking without pre-segmentation