HyKey: Hyperspectral Keypoint Detection and Matching in Minimally Invasive Surgery

2026-04-19

AI Summary

This work addresses the limitations of conventional RGB-based methods in minimally invasive surgery, where poor texture and complex illumination hinder reliable keypoint detection and matching, thereby compromising 3D reconstruction accuracy. To overcome these challenges, the authors propose HyKey, the first approach to integrate snapshot hyperspectral imaging into this domain. HyKey employs a hybrid 3D-2D convolutional network to jointly extract spatial and spectral features, enhanced by synthetic homography augmentation and epipolar geometry constraints during training. Evaluated on a newly constructed dual-camera RGB-HSI laparoscopic dataset, HyKey achieves an average matching accuracy of 96.62% and a mean Average Accuracy (mAA) of 67.18% at a 10° threshold for pose estimation on registered RGB frames, significantly outperforming RGB-based baselines such as SuperPoint and ALIKE.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Purpose: 3D reconstruction in minimally invasive surgery (MIS) enables enhanced surgical guidance through improved visualisation, tool tracking, and augmented reality. However, traditional RGB-based keypoint detection and matching pipelines struggle with surgical challenges, such as poor texture and complex illumination. We investigate whether using snapshot hyperspectral imaging (HSI) can provide improved results on keypoint detection and matching surgical scenes. Methods: We developed HyKey, a HYperspectral KEYpoint detection and description model made up of a hybrid 3D-2D convolutional neural network that jointly extracts spatial-spectral features from HSI. The model was trained using synthetic homographic augmentation and epipolar geometry constraints on a robotically-acquired dual-camera RGB-HSI laparoscopic dataset of ex-vivo organs with calibrated camera poses. We benchmarked performance against established RGB-based methods, including SuperPoint and ALIKE. Results: Our HSI-based model outperformed RGB baselines on registered RGB frames, achieving 96.62% mean matching accuracy and 67.18% mean average accuracy at 10 degree on pose estimation, demonstrating consistent improvements across multiple evaluation metrics. Conclusion: Integrating spectral information from an HSI cube offers a promising approach for robust monocular 3D reconstruction in MIS, addressing limitations of texture-poor surgical environments through enhanced spectral-spatial feature discrimination. Our model and dataset are available at https://github.com/alexsaikia/HyKey-Hyperspectral-Keypoint-Detection

Problem

• hyperspectral imaging
• keypoint detection
• minimally invasive surgery
• 3D reconstruction
• texture-poor environments

Innovation

• hyperspectral imaging
• keypoint detection
• 3D reconstruction
• minimally invasive surgery
• spatial-spectral features