Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning

2026-03-20

AI Summary

This work addresses the challenge that conventional fine-tuning often disrupts the multimodal priors of pretrained vision-language models, thereby impairing their generalization in surgical settings. To mitigate this, the authors propose a Chain-of-Adaptation (CoA) framework, which introduces reinforcement learning into domain adaptation for surgical vision-language models for the first time. CoA employs a structured reasoning mechanism to guide knowledge injection, enabling alignment with the surgical domain while preserving the model’s general-purpose vision-language understanding capabilities. Experimental results demonstrate that CoA significantly outperforms supervised fine-tuning on both in-distribution and out-of-distribution surgical data, achieving higher accuracy and behavioral stability while effectively maintaining core multimodal reasoning abilities.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Conventional fine-tuning on domain-specific datasets can inadvertently alter a model's pretrained multimodal priors, leading to reduced generalization. To address this, we propose Chain-of-Adaptation (CoA), an adaptation framework designed to integrate domain knowledge while maintaining the model's inherent reasoning and perceptual capabilities. CoA introduces a structured reasoning format that enhances domain alignment without sacrificing general multimodal competence by reinforcement learning. Experiments on standard surgical benchmarks, under both in-distribution and out-of-distribution settings, demonstrate that CoA achieves higher accuracy, stronger generalization, and more stable behavior than supervised fine-tuning. Furthermore, ablation studies confirm that CoA effectively preserves the model's core visual-language abilities, providing a reliable pathway for domain specialization in VLMs.

Problem

• domain adaptation
• vision-language models
• surgical vision
• multimodal priors
• generalization

Innovation

• Chain-of-Adaptation
• Vision-Language Models
• Reinforcement Learning
• Domain Adaptation
• Surgical AI