A Lightweight Fiducial-Based Pipeline for 3D Hyperspectral Mapping of ex-vivo Lumpectomy Specimens

2026-06-12

AI Summary

This work proposes a fully automatic, calibration-free, and lightweight method to accurately map two-dimensional hyperspectral images onto the three-dimensional geometric surface of excised breast specimens for precise intraoperative localization of suspicious tumor margins. By integrating consumer-grade RGB imaging with top-down hyperspectral acquisition, the approach employs a customized bundle adjustment constrained by ArUco marker corner points to stabilize 3D reconstruction and directly establishes correspondence between hyperspectral data and 3D geometry without estimating the hyperspectral camera pose. Combining deep learning–enhanced structured-light structure-from-motion, planar homography transformation, and orthographic depth mapping, the system achieves median 3D registration errors below 1 mm and 2D reprojection errors under 0.02 mm on two specimen cases, with processing time under four minutes per specimen.

Abstract

Hyperspectral Imaging (HSI) is a promising modality for intraoperative assessment of resection margins in Breast-Conserving Surgery (BCS), but its clinical translation requires aligning the inherently 2D spectral information onto the 3D shape of the excised tissue so that suspicious regions can be precisely localized for targeted follow-up. We present a fully automated, calibration-free pipeline that produces a 3D hyperspectral point cloud of an ex-vivo lumpectomy specimen from a set of consumer-camera RGB images and a single top-down HSI acquisition. The 3D geometry is reconstructed with a deep-learning Structure-from-Motion backbone, stabilized in a metric reference frame by a custom bundle adjustment that enforces consistency on the corners of four ArUco markers placed around the specimen. The HSI cube is then registered to the reconstruction without recovering the HSI camera pose: the markers, visible in both modalities, define 16 corner correspondences that drive a planar homography, and 3D coordinates are recovered by lookup on an orthographically rendered depth map. Evaluated on two ex-vivo lumpectomy specimens, the pipeline achieves a median 3D registration error below 1~mm and a 2D reprojection error below 0.02 mm, with a total per-specimen processing time under 4 minutes on accelerated hardware. These results support the feasibility of integrating HSI-guided spatial localization into intraoperative margin assessment workflows for breast-conserving surgery.

Problem

• 3D hyperspectral mapping
• resection margin assessment
• Breast-Conserving Surgery
• ex-vivo specimen
• spatial localization

Innovation

• hyperspectral imaging
• 3D reconstruction
• ArUco fiducials
• structure-from-motion
• image registration