Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Composition

2026-03-25

AI Summary

This work addresses the challenges posed by heterogeneous client language distributions and uneven resource availability in multilingual large language models under federated learning, which adversely impact model performance, fairness, and training efficiency. The authors extend the FederatedScope-LLM framework to support multilingual instruction tuning and introduce LDES-FL, a client-adaptive early stopping mechanism that dynamically adjusts local training based on validation performance. Their analysis reveals that intra-client multilingualism is a critical design factor: enhancing local multilingual diversity significantly improves global model performance and fairness—particularly for low-resource languages—narrowing the gap with centralized fine-tuning, albeit at the cost of increased optimization steps. In contrast, monolingual fine-tuning remains more effective for language-specific tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Federated Learning (FL) of Large Language Models (LLMs) in multilingual environments presents significant challenges stemming from heterogeneous language distributions across clients and disparities in language resource availability. To address these challenges, we extended the FederatedScope-LLM framework to support multilingual instruction-tuning experiments with LLMs. We also introduced a novel client-specific early stopping mechanism, Local Dynamic Early Stopping (LDES-FL), which allows clients to pause and resume local training based on client-side validation performance, enhancing training efficiency and sustainability. Through a series of experiments, we studied how client language composition - from fully monolingual to increasingly multilingual clients - affects multilingual quality, fairness and training cost. Monolingual local fine-tuning remains the most effective for single-language specialization, whereas federated training is better suited to learning a single balanced multilingual model. In FL, increasing within-client multilinguality leads to stronger and fairer global models, narrows the gap to centralized multilingual fine-tuning, and yields the largest gains for lower-resource languages, albeit at the cost of more optimization steps. Overall, our results identify client language composition as a key design variable in multilingual FL, shaping performance, fairness and efficiency

Problem

• Federated Learning
• Large Language Models
• Multilingual
• Language Composition
• Client Heterogeneity

Innovation

• Federated Learning
• Multilingual LLMs
• Client Language Composition
• Local Dynamic Early Stopping
• Instruction Tuning