Med-LEGO: Editing and Adapting toward Generalist Medical Image Diagnosis

2025-03-03

AI Summary

To address the challenge of cross-domain pretraining and updating general-purpose medical AI models under stringent privacy constraints, this paper proposes Med-LEGO—a training-free, editable framework for assembling medical imaging diagnosis models. Med-LEGO innovatively integrates singular value decomposition (SVD) into low-rank adaptation (LoRA), enabling efficient extraction and linear fusion of knowledge from multiple specialty-specific models without accessing original data or performing full-model retraining. It supports dynamic addition or removal of diagnostic capabilities, thereby overcoming key bottlenecks in constructing privacy-compliant generalist models. Experiments demonstrate that Med-LEGO achieves state-of-the-art performance both cross-domain and within-domain, using only 0.18% of the full model’s parameters. It significantly accelerates convergence and improves generalization to novel tasks compared to existing approaches.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The adoption of visual foundation models has become a common practice in computer-aided diagnosis (CAD). While these foundation models provide a viable solution for creating generalist medical AI, privacy concerns make it difficult to pre-train or continuously update such models across multiple domains and datasets, leading many studies to focus on specialist models. To address this challenge, we propose Med-LEGO, a training-free framework that enables the seamless integration or updating of a generalist CAD model by combining multiple specialist models, similar to assembling LEGO bricks. Med-LEGO enhances LoRA (low-rank adaptation) by incorporating singular value decomposition (SVD) to efficiently capture the domain expertise of each specialist model with minimal additional parameters. By combining these adapted weights through simple operations, Med-LEGO allows for the easy integration or modification of specific diagnostic capabilities without the need for original data or retraining. Finally, the combined model can be further adapted to new diagnostic tasks, making it a versatile generalist model. Our extensive experiments demonstrate that Med-LEGO outperforms existing methods in both cross-domain and in-domain medical tasks while using only 0.18% of full model parameters. These merged models show better convergence and generalization to new tasks, providing an effective path toward generalist medical AI.

Problem

• Enables seamless integration of specialist models for generalist medical AI.
• Addresses privacy concerns by avoiding pre-training or retraining with original data.
• Improves cross-domain and in-domain medical task performance with minimal parameters.

Innovation

• Training-free framework for generalist medical AI
• Combines specialist models using SVD-enhanced LoRA
• Enables easy integration without original data retraining