Charting Empirical Laws for LLM Fine-Tuning in Scientific Multi-Discipline Learning

2026-02-11

AI Summary

This study addresses the lack of systematic understanding of learning dynamics and generalization mechanisms in fine-tuning large language models (LLMs) across multidisciplinary scientific domains. By constructing a corpus spanning five scientific disciplines, the work systematically evaluates full-parameter fine-tuning, LoRA, LoRA-MoE, and their hybrid strategies. It introduces four empirical principles—Balance-then-Diversity, Merge-then-Align, Optimize-then-Scale, and Share-then-Specialize—combined with diversity-aware upsampling and an asymmetric low-rank mixture-of-experts architecture. Experiments reveal high variability in multidisciplinary fine-tuning outcomes and demonstrate that the proposed approach significantly enhances performance on low-resource disciplines with minimal trainable parameters, achieving robust cross-disciplinary synergies. This framework offers both theoretical insights and practical guidance for efficient generalization of scientific LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

While large language models (LLMs) have achieved strong performance through fine-tuning within individual scientific domains, their learning dynamics in multi-disciplinary contexts remains poorly understood, despite the promise of improved generalization and broader applicability through cross-domain knowledge synergy. In this work, we present the first systematic study of multi-disciplinary LLM fine-tuning, constructing a five-discipline corpus and analyzing learning patterns of full fine-tuning, LoRA, LoRA-MoE, and LoRA compositions. Particularly, our study shows that multi-disciplinary learning is substantially more variable than single-discipline training and distills four consistent empirical laws: (1) Balance-then-Diversity: low-resource disciplines degrade performance unless mitigated via diversity-aware upsampling; (2) Merge-then-Align: restoring instruction-following ability is critical for cross-discipline synergy; (3) Optimize-then-Scale: parameter scaling offers limited gains without prior design optimization; and (4) Share-then-Specialize: asymmetric LoRA-MoE yields robust gains with minimal trainable parameters via shared low-rank projection. Together, these laws form a practical recipe for principled multi-discipline fine-tuning and provide actionable guidance for developing generalizable scientific LLMs.

Problem

• multi-disciplinary learning
• LLM fine-tuning
• cross-domain knowledge synergy
• scientific generalization
• learning dynamics

Innovation

• multi-disciplinary fine-tuning
• empirical laws
• LoRA-MoE
• low-resource adaptation
• instruction alignment