Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA

2026-06-17

AI Summary

This study addresses the lack of systematic evaluation of effective adaptation strategies for French medical large language models. Focusing on French medical question answering, it is the first to disentangle the effects of adaptation methods from base model choices, systematically comparing continual pretraining (CPT), supervised fine-tuning (SFT), and their combination across diverse architectures, scales, and initialization conditions. Evaluation integrates both automatic metrics and LLM-as-a-Judge assessments. Results show that SFT offers superior cost-effectiveness on multiple-choice questions, while CPT improves text overlap metrics in open-ended generation tasks, and the combined CPT+SFT approach is consistently preferred by LLM evaluators. The work further demonstrates that French-adapted models effectively transfer to English benchmarks, providing practical adaptation guidelines for resource-constrained settings.

Abstract

The development of large language models (LLMs) has led to an increased focus on their adaptation to specialized domains and languages, yet the effectiveness of domain adaptation strategies remains unclear. We present a study of medical domain adaptation using French medical question-answering (QA) as a case study. We compare continual pretraining (CPT), supervised fine-tuning (SFT), and their combination across three model families, multiple sizes, and three initialization types, explicitly disentangling adaptation effects from base model choice. We evaluate both multiple-choice (MCQA) and open-ended QA (OEQA) under greedy and constrained decoding using automatic metrics and LLM-as-a-Judge evaluation. For MCQA, CPT+SFT most often achieves the best scores, but gains over SFT are small and frequently not statistically significant, making SFT a strong and cost-effective default. For OEQA, CPT consistently improves overlap-based metrics, while SFT often degrades generation quality; instruction tuning and CPT+SFT are preferred by LLM-based evaluation. Cross-lingual experiments further show effective transfer from French adaptation to English benchmarks. Overall, we provide practical guidelines for selecting adaptation strategies under computational constraints.

Problem

• medical LLM adaptation
• domain adaptation
• French QA
• continual pretraining
• supervised fine-tuning

Innovation

• medical LLM adaptation
• continual pretraining
• supervised fine-tuning
• cross-lingual transfer
• LLM-as-a-Judge