X-RAFT: Cross-Modal Non-Rigid Registration of Blue and White Light Neurosurgical Hyperspectral Images

2025-07-10

AI Summary

In fluorescence-guided neurosurgery, non-rigid cross-modal registration between blue-light and white-light hyperspectral images remains challenging, hindering real-time quantitative intraoperative fluorescence analysis. To address this, we propose X-RAFT—a novel end-to-end dense registration framework built upon the RAFT optical flow architecture. X-RAFT features a dual-branch cross-modal encoder, incorporates a Recursive All-Pair Field Transform for enhanced feature alignment, and introduces a self-supervised cyclic optical flow consistency loss, eliminating the need for ground-truth registration labels. Quantitative evaluation shows that X-RAFT reduces average registration error by 36.6% over baseline methods and by 27.83% relative to CrossRAFT. The model significantly improves pixel-level cross-modal correspondence accuracy and robustness under illumination and spectral variations. This enables reliable geometric foundation for intraoperative fluorescence intensity normalization and quantitative decision support, demonstrating strong clinical applicability.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Integration of hyperspectral imaging into fluorescence-guided neurosurgery has the potential to improve surgical decision making by providing quantitative fluorescence measurements in real-time. Quantitative fluorescence requires paired spectral data in fluorescence (blue light) and reflectance (white light) mode. Blue and white image acquisition needs to be performed sequentially in a potentially dynamic surgical environment. A key component to the fluorescence quantification process is therefore the ability to find dense cross-modal image correspondences between two hyperspectral images taken under these drastically different lighting conditions. We address this challenge with the introduction of X-RAFT, a Recurrent All-Pairs Field Transforms (RAFT) optical flow model modified for cross-modal inputs. We propose using distinct image encoders for each modality pair, and fine-tune these in a self-supervised manner using flow-cycle-consistency on our neurosurgical hyperspectral data. We show an error reduction of 36.6% across our evaluation metrics when comparing to a naive baseline and 27.83% reduction compared to an existing cross-modal optical flow method (CrossRAFT). Our code and models will be made publicly available after the review process.

Problem

• Cross-modal registration of blue and white neurosurgical images
• Quantitative fluorescence needs paired spectral data
• Dense correspondences between different lighting condition images

Innovation

• Modified RAFT model for cross-modal optical flow
• Self-supervised fine-tuning with flow-cycle-consistency
• Distinct encoders for blue and white light images