Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery

2026-06-22

AI Summary

This work addresses the limitations of existing robot-assisted surgical error detection methods, which often neglect fine-grained procedural context and underutilize multimodal information. To overcome these challenges, the authors propose a unified activity-aware multimodal framework that, for the first time, deeply integrates structured textual prompts, video streams, and kinematic data. The approach leverages a pretrained visual encoder, contrastive language–image embeddings, and an activity prompting mechanism to achieve contextual awareness at multiple levels—gesture-level activities, instrument–object interactions, and error definitions. Evaluated on the JIGSAWS and SAR-RARP50 datasets, the method achieves F1 score improvements of 5% and 16.6%, respectively, substantially outperforming current state-of-the-art approaches.

Abstract

Robot-assisted minimally invasive surgery improves surgical precision but introduces complexity, making technical error detection essential for ensuring patient safety. Current executional error detection methods using video data often overlook fine-grained contextual descriptions of activities and error types within the hierarchical structure of surgical procedures. They also under-utilize complementary multimodal information. We propose a unified framework for executional error detection that leverages multimodal input, including video, kinematics, and descriptive textual prompts. Through activity prompting, we integrate descriptive language in gesture-level activities, instrument-object interactions, and error definitions. We also introduce activity-aware visual embeddings derived from vision encoders pretrained on surgical activity labels to compare the effectiveness of contrastive language-image embeddings with traditional image-based embeddings for error detection. By seamlessly integrating kinematic data with video and textual modalities, our framework significantly improves error detection performance. Achieving up to 5\% and 16.6\% F1 score improvements over state-of-the-art baselines on the JIGSAWS and SAR-RARP50 datasets, respectively, we demonstrate the value of combining curated textual prompts with multimodal data for accurate error detection.

Problem

• error detection
• robot-assisted surgery
• multimodal learning
• surgical activity understanding
• real-time monitoring

Innovation

• multimodal fusion
• activity-aware embedding
• executional error detection
• textual prompting
• robot-assisted surgery