EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery

2025-06-07

AI Summary

To address cross-task interference in laparoscopic surgery caused by high activity variability and target-background ambiguity, this work proposes a multi-task collaborative learning framework that jointly enhances surgical activity recognition and organ/instrument semantic segmentation. We introduce Task-Efficient Shared Low-Rank Adapters to mitigate gradient conflicts and Spatially-Aware Multi-Scale Attention to strengthen spatial discrimination. Additionally, we construct the first end-to-end, dual-annotated laparoscopic multi-task dataset family—MTLESD, MTLEndovis, and MTLEndovis-Gen. Building upon DINOv2, our approach integrates LoRA fine-tuning, low-rank shared adapters, and joint optimization. Evaluated on three novel benchmarks, it achieves state-of-the-art performance: +5.2% accuracy in activity recognition and +4.8% mIoU in segmentation, with markedly improved robustness and generalization—establishing a new paradigm for real-time, safety-critical AI-assisted surgery.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Endoscopic surgery is the gold standard for robotic-assisted minimally invasive surgery, offering significant advantages in early disease detection and precise interventions. However, the complexity of surgical scenes, characterized by high variability in different surgical activity scenarios and confused image features between targets and the background, presents challenges for surgical environment understanding. Traditional deep learning models often struggle with cross-activity interference, leading to suboptimal performance in each downstream task. To address this limitation, we explore multi-task learning, which utilizes the interrelated features between tasks to enhance overall task performance. In this paper, we propose EndoARSS, a novel multi-task learning framework specifically designed for endoscopy surgery activity recognition and semantic segmentation. Built upon the DINOv2 foundation model, our approach integrates Low-Rank Adaptation to facilitate efficient fine-tuning while incorporating Task Efficient Shared Low-Rank Adapters to mitigate gradient conflicts across diverse tasks. Additionally, we introduce the Spatially-Aware Multi-Scale Attention that enhances feature representation discrimination by enabling cross-spatial learning of global information. In order to evaluate the effectiveness of our framework, we present three novel datasets, MTLESD, MTLEndovis and MTLEndovis-Gen, tailored for endoscopic surgery scenarios with detailed annotations for both activity recognition and semantic segmentation tasks. Extensive experiments demonstrate that EndoARSS achieves remarkable performance across multiple benchmarks, significantly improving both accuracy and robustness in comparison to existing models. These results underscore the potential of EndoARSS to advance AI-driven endoscopic surgical systems, offering valuable insights for enhancing surgical safety and efficiency.

Problem

• Addressing high variability in endoscopic surgical activity scenarios
• Overcoming confused image features between targets and background
• Mitigating cross-activity interference in multi-task surgical understanding

Innovation

• Uses DINOv2 foundation model with Low-Rank Adaptation
• Incorporates Task Efficient Shared Low-Rank Adapters
• Introduces Spatially-Aware Multi-Scale Attention mechanism