Distilling Expert Surgical Knowledge: How to train local surgical VLMs for anatomy explanation in Complete Mesocolic Excision

2025-12-05

AI Summary

Current visual language models (VLMs) exhibit limited capability in identifying and interpreting anatomical landmarks during complete mesocolic excision (CME), while their reliance on cloud-based deployment raises significant patient data privacy concerns. To address these challenges, this work proposes a privacy-preserving, on-device surgical scene understanding framework. Our method generates high-fidelity, de-identified synthetic training data solely from textual prompts and binary segmentation masks—eliminating the need for real intraoperative imagery. Domain adaptation is achieved by jointly leveraging spatial mask inputs and large-model-generated textual context through supervised fine-tuning (SFT) and direct preference optimization (DPO). Experimental results demonstrate substantial performance gains for local VLMs on CME anatomical understanding tasks, achieving high data efficiency, rigorous privacy protection, and clinical deployability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Recently, Vision Large Language Models (VLMs) have demonstrated high potential in computer-aided diagnosis and decision-support. However, current VLMs show deficits in domain specific surgical scene understanding, such as identifying and explaining anatomical landmarks during Complete Mesocolic Excision. Additionally, there is a need for locally deployable models to avoid patient data leakage to large VLMs, hosted outside the clinic. We propose a privacy-preserving framework to distill knowledge from large, general-purpose LLMs into an efficient, local VLM. We generate an expert-supervised dataset by prompting a teacher LLM without sensitive images, using only textual context and binary segmentation masks for spatial information. This dataset is used for Supervised Fine-Tuning (SFT) and subsequent Direct Preference Optimization (DPO) of the locally deployable VLM. Our evaluation confirms that finetuning VLMs with our generated datasets increases surgical domain knowledge compared to its base VLM by a large margin. Overall, this work validates a data-efficient and privacy-conforming way to train a surgical domain optimized, locally deployable VLM for surgical scene understanding.

Problem

• Train local surgical VLMs to explain anatomy during Complete Mesocolic Excision
• Address privacy concerns by avoiding patient data leakage to external VLMs
• Distill expert surgical knowledge from large LLMs into efficient local models

Innovation

• Distilling knowledge from large LLMs into local VLM
• Generating dataset using text and segmentation masks
• Fine-tuning local VLM with SFT and DPO methods