SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy

2026-07-05

AI Summary

This work formalizes surgical affordance prediction as a visual task for the first time, aiming to identify visually discernible regions amenable to fundamental surgical actions and thereby bridge the perception-action gap in surgical automation. To this end, the authors propose an adaptive multimodal feature fusion framework that integrates a self-supervised vision transformer with a large-scale generative model encoder, augmented by hierarchical prompt learning and a scene-guided attention decoder to effectively combine semantic understanding with spatial awareness. Evaluated on a newly curated dataset annotated for affordances across three basic surgical maneuvers, the proposed method achieves state-of-the-art performance and successfully enables autonomous robotic execution of surgical tasks on lung and prostate phantoms.

Abstract

Surgical automation is being increasingly studied, yet bridging visual scene understanding with autonomous action planning remains a fundamental challenge. While much research effort has been made on scene perception (e.g., tool recognition and scene segmentation), understanding and predicting actionable possibilities for surgical automation is still underexplored. In this paper, we introduce surgical affordance prediction, which identifies actionable regions for fundamental surgical actions from visual data. Specifically, a novel adaptive feature fusion framework is proposed that leverages the complementary strengths of a self-supervised vision transformer encoder for its superior semantic understanding and a large-scale generative model encoder for its spatially-aware capability. Furthermore, we introduce a hierarchical prompt learning mechanism to adapt to varying procedural contexts. Finally, a scene-guided attention decoder is proposed to focus on critical surgical areas while suppressing background distractions. To validate the effectiveness, we established a new dataset, derived from publicly available surgical datasets with affordance annotations for three basic surgical actions: aspiration, clipping, and retraction. Extensive experiments demonstrate that our approach achieves state-of-the-art performance. Moreover, we validate our framework's applicability for downstream automation on a realistic lung and prostate phantom, and results show that the predicted affordance maps successfully enable autonomous surgical actions.

Problem

• surgical affordance prediction
• robot autonomy
• actionable regions
• surgical automation
• visual scene understanding

Innovation

• surgical affordance prediction
• multimodal feature fusion
• adaptive prompt learning
• scene-guided attention
• robotic autonomy