Evaluating and Understanding Model Editing for Medical Vision Language Models

2026-07-06

AI Summary

Existing editing methods for medical vision-language models (VLMs) lack evaluation benchmarks grounded in real-world clinical scenarios, hindering reliable assessment of their fidelity, precision, and generalization. This work proposes M3Bench—the first clinically oriented benchmark for editing multimodal medical models—comprising 16,276 questions spanning diverse anatomical structures, imaging modalities, and clinical specialties, and supporting both single-step and sequential editing. We systematically evaluate four classes of editors across six VLMs, uncovering critical limitations in locality, compositional generalization, and cross-modal robustness. These shortcomings are linked to the geometric structure of the VLMs’ latent spaces: gradient-based methods exhibit strong generalization but compromise locality, whereas memory-based approaches preserve locality yet suffer from weak generalization and high sensitivity to backbone architectures. Our findings offer empirical guidance for the safe and reliable deployment of post-hoc editing in clinical settings.

Abstract

Model editing promises a fast, targeted way to correct post-deployment mistakes in medical vision-language models (VLMs) without costly retraining. However, existing multimodal model editing benchmarks focus on general-purpose tasks and do not reflect realistic clinical domain requirements and variability. To address this, we introduce M3Bench, a clinically grounded benchmark for multimodal model editing that evaluates whether an edit remains reliable, precise, and generalizable under the challenges of image and text variation, modality and protocol shifts, clinical knowledge composition, and temporal progression. M3Bench contains 16,276 questions spanning diverse anatomy, modalities, and specialties, and supports both single and sequential edits. By evaluating 4 representative editors across 6 medical and general VLMs, we find that no method excels across all criteria. Gradient-based editors achieve strong transfer but suffer from catastrophic locality violations, whereas memory-based methods preserve locality but lack compositional generality and exhibit high backbone-dependent hyperparameter sensitivity. We further attribute these failures to the latent space geometry of VLMs and how different editing methods shift its landscape. Overall, M3Bench establishes a rigorous clinical stress test for multimodal model editing and offers actionable guidance for safer post-deployment adaptation. The benchmark is publicly available at https://github.com/BioMed-AI-Lab-U-Michgan/M3Bench .

Problem

• model editing
• medical vision-language models
• multimodal benchmark
• clinical evaluation
• post-deployment adaptation

Innovation

• model editing
• medical vision-language models
• multimodal benchmark
• clinical generalization
• latent space geometry