Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus

2026-04-08

AI Summary

This study addresses the challenge of specializing large language models for biomedicine in non-English contexts, with a focus on French. It employs domain-adaptive pretraining (DAPT) to continually pretrain small-to-medium-scale French language models and constructs the first commercially usable, openly licensed French health corpus. Through rigorous data curation, causal language modeling, and model merging techniques, the work systematically evaluates DAPT’s efficacy and the associated degradation of general-purpose capabilities under resource-constrained settings. Results demonstrate that while DAPT is feasible at smaller scales, its benefits are limited; however, model merging not only mitigates the loss of general performance but can also enhance downstream task results. The released corpus and specialized models establish a new paradigm for domain adaptation in low-resource languages.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains, yet their adaptation to specialized fields remains challenging, particularly for non-English languages. This study investigates domain-adaptive pre-training (DAPT) as a strategy for specializing small to mid-sized LLMs in the French biomedical domain through continued pre-training. We address two key research questions: the viability of specialized continued pre-training for domain adaptation and the relationship between domain-specific performance gains and general capability degradation. Our contributions include the release of a fully open-licensed French biomedical corpus suitable for commercial and open-source applications, the training and release of specialized French biomedical LLMs, and novel insights for DAPT implementation. Our methodology encompasses the collection and refinement of high-quality French biomedical texts, the exploration of causal language modeling approaches using DAPT, and conducting extensive comparative evaluations. Our results cast doubt on the efficacy of DAPT, in contrast to previous works, but we highlight its viability in smaller-scale, resource-constrained scenarios under the right conditions. Findings in this paper further suggest that model merging post-DAPT is essential to mitigate generalization trade-offs, and in some cases even improves performance on specialized tasks at which the DAPT was directed.

Problem

• domain adaptation
• biomedical language modeling
• French LLMs
• continued pre-training
• specialization trade-offs

Innovation

• domain-adaptive pre-training
• French biomedical corpus
• model merging
• continued pre-training
• specialized LLMs