Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI

2025-04-28

AI Summary

To address the need for automated summarization of surgical videos, this paper proposes an end-to-end framework integrating a multimodal vision–temporal Transformer with generative AI. Methodologically, it employs a three-tier cascaded architecture: (1) a Vision Transformer (ViT) extracts frame-level visual features; (2) a Video Vision Transformer (ViViT) models temporal dynamics of surgical actions; and (3) a domain-adapted large language model (LLM) jointly performs surgical tool detection, action understanding, and clinical report generation. The key contributions include the first unified integration of multi-granularity perception, temporal modeling, and structured report generation within a single pipeline, along with novel multimodal feature alignment and hierarchical summarization mechanisms. Evaluated on the CholecT50 dataset, the framework achieves 96% accuracy in tool detection and a BERTScore of 0.74 for temporal summarization—both significantly surpassing state-of-the-art methods.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The automatic summarization of surgical videos is essential for enhancing procedural documentation, supporting surgical training, and facilitating post-operative analysis. This paper presents a novel method at the intersection of artificial intelligence and medicine, aiming to develop machine learning models with direct real-world applications in surgical contexts. We propose a multi-modal framework that leverages recent advancements in computer vision and large language models to generate comprehensive video summaries. % The approach is structured in three key stages. First, surgical videos are divided into clips, and visual features are extracted at the frame level using visual transformers. This step focuses on detecting tools, tissues, organs, and surgical actions. Second, the extracted features are transformed into frame-level captions via large language models. These are then combined with temporal features, captured using a ViViT-based encoder, to produce clip-level summaries that reflect the broader context of each video segment. Finally, the clip-level descriptions are aggregated into a full surgical report using a dedicated LLM tailored for the summarization task. % We evaluate our method on the CholecT50 dataset, using instrument and action annotations from 50 laparoscopic videos. The results show strong performance, achieving 96% precision in tool detection and a BERT score of 0.74 for temporal context summarization. This work contributes to the advancement of AI-assisted tools for surgical reporting, offering a step toward more intelligent and reliable clinical documentation.

Problem

• Automating surgical video summarization for better documentation
• Developing AI models for real-world surgical applications
• Generating comprehensive reports from multimodal surgical data

Innovation

• Multimodal visual-temporal transformers for surgical video analysis
• Generative AI for comprehensive surgical video summaries
• LLM-based aggregation for full surgical report generation