A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine

2026-01-29

AI Summary

This work addresses the limited generalization and safety of large medical language models trained on single-institution data, as well as the challenges posed by high communication costs and clinical data heterogeneity in conventional federated learning. To overcome these issues, the authors propose Fed-MedLoRA and its enhanced variant Fed-MedLoRA+, a model-agnostic, parameter-efficient federated learning framework that transmits only low-rank adapter (LoRA) parameters. The approach incorporates a data-aware adaptive aggregation mechanism to mitigate cross-institutional heterogeneity. Experimental results across five patient cohorts demonstrate that the proposed method substantially reduces communication and computational overhead while consistently outperforming baseline models—including BERT, LLaMA-3, and DeepSeek-R1—in in-domain evaluation, external validation, and adaptation to low-resource new sites.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis. To enable their use in clinical settings, LLMs are typically further adapted through continued pretraining or post-training using clinical data. However, most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems. Federated learning (FL) is a promising solution for enabling collaborative model development across healthcare institutions. Yet applying FL to LLMs in medicine remains fundamentally limited. First, conventional FL requires transmitting the full model during each communication round, which becomes impractical for multi-billion-parameter LLMs given the limited computational resources. Second, many FL algorithms implicitly assume data homogeneity, whereas real-world clinical data are highly heterogeneous across patients, diseases, and institutional practices. We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications. Fed-MedLoRA transmits only low-rank adapter parameters, reducing communication and computation overhead, while Fed-MedLoRA+ further incorporates adaptive, data-aware aggregation to improve convergence under cross-site heterogeneity. We apply the framework to clinical information extraction (IE), which transforms patient narratives into structured medical entities and relations. Accuracy was assessed across five patient cohorts through comparisons with BERT models, and LLaMA-3 and DeepSeek-R1, GPT-4o models. Evaluation settings included (1) in-domain training and testing, (2) external validation on independent cohorts, and (3) a low-resource new-site adaptation scenario using real-world clinical notes from the Yale New Haven Health System.

Problem

• federated learning
• large language models
• medical data heterogeneity
• parameter efficiency
• clinical information extraction

Innovation

• Federated Learning
• Parameter-Efficient Fine-Tuning
• Medical Large Language Models
• Low-Rank Adaptation
• Data Heterogeneity