Dense Structural Priors for Sparse Functional Landmark Localization in Surgical Videos

2026-06-29

AI Summary

Existing vision foundation models struggle to accurately localize functional keypoints—such as instrument tips and anchors—that are semantically tied to surgical actions like grasping or clamping in surgical videos. To address this, this work proposes a lightweight multi-frame network that leverages zero-shot point-prompt masks generated by SAM³ as dense structural priors. By fusing visual features with heatmap regression, the method achieves precise keypoint localization without requiring manual pixel-level annotations. The approach mitigates region bias inherent in direct mask supervision by using vision foundation model outputs as action-aware structural guidance. Evaluated across 7,867 surgical clips from five diverse datasets, the method achieves F1 scores of 72.4% for tip localization and 58.0% for anchor localization, demonstrating its effectiveness in heterogeneous surgical scenarios.

Abstract

Vision foundation models such as SAM 3 can provide transferable object-level structure across diverse surgical video conditions, but segmentation outputs do not explicitly encode the action-conditioned semantics that define functional surgical landmarks. Estimating instrument extent and geometry differs from localizing the tip or anchor relevant to clipping, grasping, or dissecting. We investigate vision foundation model-enabled sparse action-aware landmark localization, using zero-shot, point-prompted structural masks to provide dense instrument-level context without manual pixel-level mask annotations. We propose a lightweight refinement framework that uses SAM 3 as a structural prior. A coarse multi-frame network predicts tip and anchor prompts, generating non-oracle masks that are fused with visual and heatmap features to refine functional landmark predictions. We compare direct mask-augmented supervision, prediction-derived mask-prior refinement, and auxiliary mask supervision to examine how vision foundation model-derived structure should enter a precision-oriented localization system. Experiments on 7,867 clips from 60 surgical videos spanning YouTube, Cholec80, HeiChole, SurgVU, and CRCD evaluate the approach under heterogeneous conditions. Without manual pixel-level mask annotations for training, the proposed model achieves overall F1 scores of 72.4% for tip and 58.0% for anchor localization. Directly imposing masks on heatmap targets biases learning toward broad tool regions, whereas prediction-derived priors and auxiliary supervision provide effective intermediate structural guidance for action-dependent landmark prediction.

Problem

• functional landmark localization
• surgical videos
• vision foundation models
• action-aware
• sparse localization

Innovation

• vision foundation models
• functional landmark localization
• structural priors
• sparse action-aware prediction
• mask-prior refinement