Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts

2024-10-14

arXiv.org

AI Summary

Large language models (LLMs) face significant challenges in multilingual medical applications for low-resource languages, including poor cross-lingual transferability and severe scarcity of high-quality annotated medical data. Method: We propose Post-MoE—a sparse mixture-of-experts architecture applied exclusively to the top layers of an LLM—motivated by “terminal divergence” in cross-lingual information flow. Experts are grouped using linguistic phylogeny priors, and a high-quality multilingual medical dataset is curated. Routing is analytically optimized via circuit-theoretic principles to ensure interpretability and zero additional parameters. Contribution/Results: Evaluated across 50 languages, our approach achieves a 23.6% average improvement in zero-shot medical question answering performance. It simultaneously delivers superior scalability, parameter efficiency, and architectural interpretability, establishing a novel paradigm for multilingual medical AI.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Adapting medical Large Language Models to local languages can reduce barriers to accessing healthcare services, but data scarcity remains a significant challenge, particularly for low-resource languages. To address this, we first construct a high-quality medical dataset and conduct analysis to ensure its quality. In order to leverage the generalization capability of multilingual LLMs to efficiently scale to more resource-constrained languages, we explore the internal information flow of LLMs from a multilingual perspective using Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE routing method that employs language-specific experts and cross-lingual routing. Inspired by circuit theory, our routing analysis revealed a Spread Out in the End information flow mechanism: while earlier layers concentrate cross-lingual information flow, the later layers exhibit language-specific divergence. This insight directly led to the development of the Post-MoE architecture, which applies sparse routing only in the later layers while maintaining dense others. Experimental results demonstrate that this approach enhances the generalization of multilingual models to other languages while preserving interpretability. Finally, to efficiently scale the model to 50 languages, we introduce the concept of language family experts, drawing on linguistic priors, which enables scaling the number of languages without adding additional parameters.

Problem

• Adapting medical LLMs to low-resource languages
• Overcoming data scarcity in multilingual medical datasets
• Scaling medical LLMs efficiently to 50 languages

Innovation

• Mixture of Language Family Experts
• Post-MoE architecture
• Sparse routing in later layers