DoMIX: An Efficient Framework for Exploiting Domain Knowledge in Fine-Tuning

2025-07-03

AI Summary

Existing continual domain-adaptive pretraining (DAP) methods suffer from three key bottlenecks: high computational and memory overhead, sensitivity to the order of incremental domain data, and generation of only a single generic model—contradicting DAP’s inherent domain-specificity. To address these, we propose DoMIX, a parallelized, parameter-efficient DAP framework built upon LoRA. Its core innovations include: (i) enabling concurrent self-supervised pretraining across multiple domains via plug-and-play LoRA modules, thereby decoupling domain-specific updates and eliminating sequential dependency; and (ii) supporting on-demand generation of lightweight, task-specialized models that jointly preserve accumulated knowledge and enable fine-grained customization. Evaluated on multi-domain benchmarks, DoMIX reduces GPU memory consumption by 42% on average and training time by 38%, while seamlessly transferring to standard large-model fine-tuning scenarios—demonstrating superior efficiency, robustness, and practicality.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Diffusion Models for Vision
• Application Domains: Security

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Domain-Adaptive Pre-training (DAP) has recently gained attention for its effectiveness in fine-tuning pre-trained models. Building on this, continual DAP has been explored to develop pre-trained models capable of incrementally incorporating different domain datasets. However, existing continual DAP methods face several limitations: (1) high computational cost and GPU memory usage during training; (2) sensitivity to incremental data order; and (3) providing a single, generalized model for all end tasks, which contradicts the essence of DAP. In this paper, we propose DoMIX, a novel approach that addresses these challenges by leveraging LoRA modules, a representative parameter-efficient fine-tuning (PEFT) method. Our approach enables efficient and parallel domain-adaptive pre-training that is robust to domain order and effectively utilizes accumulated knowledge to provide tailored pre-trained models for specific tasks. We also demonstrate that our method can be extended beyond the DAP setting to standard LLM fine-tuning scenarios. Code is available at https://github.com/dohoonkim-ai/DoMIX.

Problem

• Reduces high computational cost in continual DAP
• Minimizes sensitivity to incremental data order
• Provides task-specific models instead of generalized ones

Innovation

• Leverages LoRA modules for efficient fine-tuning
• Enables parallel domain-adaptive pre-training
• Provides tailored models for specific tasks