IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation

2025-10-23

AI Summary

To address the degradation of instruction-following capability and semantic representation during continual pretraining of large language models (LLMs) on new domains, this paper proposes IKnow—a novel framework that requires neither the original base model weights nor external domain-specific knowledge bases. IKnow introduces an instruction-response dialogue-based self-supervised objective, enabling deep semantic encoding by explicitly uncovering implicit domain knowledge embedded in raw text. Its core innovation lies in jointly modeling instruction awareness and knowledge awareness, thereby simultaneously enhancing domain adaptability and instruction-following robustness under fully unsupervised conditions. Extensive experiments across multiple domain-transfer tasks demonstrate that IKnow significantly outperforms baseline methods while preserving general-purpose capabilities. These results validate its effectiveness and generalizability in resource-constrained real-world deployment scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Continual pretraining promises to adapt large language models (LLMs) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to instruction-tuned models is known to degrade their instruction-following capability and semantic representations. Existing fixes assume access to the original base model or rely on knowledge from an external domain-specific database - both of which pose a realistic barrier in settings where the base model weights are withheld for safety reasons or reliable external corpora are unavailable. In this work, we propose Instruction-Knowledge-Aware Continual Adaptation (IKnow), a simple and general framework that formulates novel self-supervised objectives in the instruction-response dialogue format. Rather than depend- ing on external resources, IKnow leverages domain knowledge embedded within the text itself and learns to encode it at a deeper semantic level.

Problem

• Adapts LLMs to new domains using unlabeled data only
• Preserves instruction-following ability during continual pretraining
• Leverages embedded domain knowledge without external resources

Innovation

• Instruction-response dialogue format objectives
• Leverages embedded domain knowledge internally
• Encodes knowledge at deeper semantic level