Continual Pre-Training is (not) What You Need in Domain Adaption

2025-04-18

AI Summary

This study investigates the impact of Domain-Adaptive Continual Pretraining (DACP) on the reasoning capabilities of large language models (LLMs) in the legal domain. Leveraging open-source architectures (e.g., LLaMA, Phi), we conduct continual pretraining and subsequent instruction tuning on Taiwanese legal corpora, and construct a multi-granularity evaluation framework covering case-based reasoning, statutory application, and logical abductive inference. Our key findings reveal that DACP substantially enhances legal terminology comprehension and domain-specific knowledge recall (+12.3%), yet degrades zero-shot prompting generalization across diverse reasoning tasks (−8.7% average accuracy). This work provides the first systematic characterization of the performance trade-offs inherent in DACP for legal LLMs, challenging the prevailing assumption that continual pretraining universally improves domain adaptation. It clarifies DACP’s applicability boundary: it benefits knowledge-intensive tasks but undermines reasoning scenarios requiring robust, general-purpose instruction following.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The recent advances in Legal Large Language Models (LLMs) have transformed the landscape of legal research and practice by automating tasks, enhancing research precision, and supporting complex decision-making processes. However, effectively adapting LLMs to the legal domain remains challenging due to the complexity of legal reasoning, the need for precise interpretation of specialized language, and the potential for hallucinations. This paper examines the efficacy of Domain-Adaptive Continual Pre-Training (DACP) in improving the legal reasoning capabilities of LLMs. Through a series of experiments on legal reasoning tasks within the Taiwanese legal framework, we demonstrate that while DACP enhances domain-specific knowledge, it does not uniformly improve performance across all legal tasks. We discuss the trade-offs involved in DACP, particularly its impact on model generalization and performance in prompt-based tasks, and propose directions for future research to optimize domain adaptation strategies in legal AI.

Problem

• Evaluating Domain-Adaptive Continual Pre-Training for legal LLMs
• Assessing DACP's impact on legal reasoning and task performance
• Exploring trade-offs between domain knowledge and model generalization

Innovation

• Domain-Adaptive Continual Pre-Training for legal LLMs
• Evaluates DACP on Taiwanese legal reasoning tasks
• Discusses trade-offs in generalization and prompt performance