CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation

2026-08-30

AI Summary

本文提出CPR方法，通过在基础模型和领域专家模型间进行关键点路由，解决大语言模型在领域适应中出现的灾难性遗忘问题。

Abstract

Supervised fine-tuning (SFT) is the de facto standard for adapting large language models (LLMs) to target domains, but it often degrades the model's general capabilities, a phenomenon known as catastrophic forgetting. Existing approaches typically modify the SFT loss to mitigate forgetting, but they inevitably operate along a domain-generality trade-off. In this work, we step outside this trade-off by decoupling the two capabilities at the model level: we keep the original base model for general capability, and selectively invoke the SFT expert only when domain-specific knowledge is required. Specifically, we propose CPR (Critical-Point Routing), a token-level routing framework between a base model and its expert derivative, based on critical tokens where the base model fails but the expert succeeds. We train a lightweight hierarchical router that estimates the expert-call probability per token, and pair it with a tailored inference procedure that combines momentum smoothing and threshold gating. Across diverse model-domain configurations, CPR achieves state-of-the-art across all settings, surpassing SFT expert by 1.4-5.5% in domain performance while recovering its general-capability drop from 3.4-14.5% to at most 0.5%, with minimal overhead from invoking the expert on only one-third of tokens.

Problem

• Catastrophic Forgetting
• Domain Adaptation
• Large Language Models
• Supervised Fine-Tuning

Innovation

• Critical-Point Routing
• token-level routing
• hierarchical router
• momentum smoothing
• threshold gating