Two-Stage Mixture-of-LoRA for Multi-Task Medical Vision-Language Learning

2026-09-13

AI Summary

本文提出两阶段混合LoRA框架解决多任务医学视觉-语言学习中的异构输出、梯度冲突及数据不平衡问题，通过共享和特定任务的LoRA架构实现。

Abstract

Medical vision-language models (VLMs) allow a single model to perform clinical image analysis tasks ranging from diagnosis classification to report generation. However, joint adaptation is challenged by heterogeneous output formats, conflicting task gradients, and imbalanced training data. Hence, we present \textbf{Two-Stage Mixture-of-LoRA}, a framework built on MedGemma-1.5-4B. The framework uses a shared-specific Mixture-of-LoRA architecture comprising one shared LoRA and six task-specific expert LoRAs, together with a two-stage training procedure. In Stage 1, we jointly train the shared LoRA and all task-specific expert LoRAs on all tasks. In Stage 2, we first freeze the backbone, the shared LoRA, and all non-target experts, and refine one task expert at a time. Classification and regression then receive an additional modality-balanced continuation, in which smaller modality groups are repeated to match the largest group. In the FLARE 2026 Task 3 test sets, the proposed method achieves 0.85 balanced accuracy for classification, 0.48 micro-F1 for multi-label classification, 0.79 detection F1, and 17.39 regression MAE. Code is available at https://github.com/YuanYL03/MICCAI-FLARE-2026-Challenge-Task3-2D.

Problem

• Medical Vision-Language Models
• Heterogeneous Output Formats
• Conflicting Task Gradients
• Imbalanced Training Data

Innovation

• Two-Stage Mixture-of-LoRA
• shared-specific Mixture-of-LoRA architecture
• two-stage training procedure
• modality-balanced continuation