DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations

2025-10-09

AI Summary

Small language models (LLMs) exhibit weak zero-shot cross-domain instruction-following capabilities, and conventional fine-tuning often induces catastrophic forgetting, degrading generalization. To address this, we propose DACIP-RC—a novel domain adaptation framework that replaces fine-tuning with continual pre-training. DACIP-RC is the first to apply instruction pre-training to commercial dialogue corpora; it automatically constructs diverse instruction-response pairs via a reading comprehension–inspired mechanism. Integrating reading-comprehension–based instruction generation, continual pre-training, and zero-shot transfer learning, DACIP-RC enables effective, forgetting-free domain adaptation. Empirical evaluation on real-world business tasks—including meeting summarization, action-item generation, and call-purpose identification—demonstrates substantial improvements in zero-shot performance for small LLMs. Results confirm DACIP-RC’s effectiveness, robustness across multiple scenarios, and industrial scalability.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: (Large) Language Models
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The rapid advancements in Large Language Models (LLMs) have enabled their adoption in real-world industrial scenarios for various natural language processing tasks. However, the high inference cost of large-scale LLMs makes their deployment impractical, necessitating the use of smaller models. Despite their efficiency, smaller LLMs lack robust zero-shot instruction-following capabilities across diverse domains, limiting their adaptability to dynamic user requirements. Traditional fine-tuning approaches exacerbate this issue by inducing catastrophic forgetting, reducing the model's generalization ability for unseen tasks. In this paper, we propose Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension (DACIP-RC), a continual pre-training technique that enhances smaller LLMs' domain adaptability for business conversational tasks. Unlike conventional pre-training approaches that rely on next-token prediction, DACIP-RC generates diverse task instructions and responses via reading comprehension on conversation transcripts, enabling better instruction generalization. Our empirical evaluations demonstrate that DACIP-RC significantly improves zero-shot generalization across a wide range of business conversational tasks, including meeting summarization, action item generation, and call purpose identification. To the best of our knowledge, this is the first work to apply instruction pre-training on business conversational data, providing insights into how industries can leverage proprietary datasets for domain adaptation.

Problem

• Smaller LLMs lack robust zero-shot instruction-following across domains
• Traditional fine-tuning causes catastrophic forgetting in smaller models
• Limited adaptability of smaller LLMs to dynamic business requirements

Innovation

• Domain adaptive continual pre-training for small LLMs
• Generates task instructions via reading comprehension
• Enhances zero-shot generalization for business conversations