From Phase Grounding to Intelligent Surgical Narratives

2026-03-05

AI Summary

Traditional postoperative reports are often vague, and manual annotation of surgical videos is prohibitively expensive, highlighting a critical need for efficient methods to generate structured surgical timelines and narratives. This work proposes a CLIP-based multimodal framework that, for the first time, leverages pretrained multimodal representations to enable end-to-end alignment between surgical video frames and textual descriptions. By establishing fine-grained correspondences between visual frames and gesture semantics within a shared embedding space, the model automatically predicts surgical phases and actions directly from video inputs. The approach substantially reduces reliance on costly manual annotations and offers a practical pathway toward building intelligent, accurate, and efficient surgical narrative systems.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Video surgery timelines are an important part of tool-assisted surgeries, as they allow surgeons to quickly focus on key parts of the procedure. Current methods involve the surgeon filling out a post-operation (OP) report, which is often vague, or manually annotating the surgical videos, which is highly time-consuming. Our proposed method sits between these two extremes: we aim to automatically create a surgical timeline and narrative directly from the surgical video. To achieve this, we employ a CLIP-based multi-modal framework that aligns surgical video frames with textual gesture descriptions. Specifically, we use the CLIP visual encoder to extract representations from surgical video frames and the text encoder to embed the corresponding gesture sentences into a shared embedding space. We then fine-tune the model to improve the alignment between video gestures and textual tokens. Once trained, the model predicts gestures and phases for video frames, enabling the construction of a structured surgical timeline. This approach leverages pretrained multi-modal representations to bridge visual gestures and textual narratives, reducing the need for manual video review and annotation by surgeons.

Problem

• surgical video
• timeline generation
• automatic annotation
• surgical narrative
• phase grounding

Innovation

• CLIP-based multimodal framework
• surgical video understanding
• automatic surgical timeline
• gesture-phase alignment
• intelligent surgical narrative