When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization

2026-08-29

AI Summary

为解决领域特化时保持通用能力的问题，提出Engram Adapter框架，通过条件性记忆适配器选择性地对输入进行调整，提高领域内性能同时保留99.4%-100.1%的领域外性能。

Abstract

Large language models deployed in specialized domains must improve in-domain performance without sacrificing general capabilities. Existing parameter-efficient fine-tuning methods are typically always on: their learned perturbations are applied to every input, which can degrade out-of-domain (OOD) performance. We propose Engram Adapter, a framework that repurposes pretraining-time conditional memory as a post-hoc adapter for frozen LLMs. It uses multi-channel matching over local n-gram patterns with explicit occupancy tracking as a lightweight selectivity prior, making residual injection more likely on in-domain inputs while a learned scalar gate suppresses incoherent OOD retrievals. We evaluate on Qwen3-4B and Qwen3-8B with AG-News and MedMCQA as adaptation tasks and OOD benchmarks spanning reasoning, translation, code generation, and legal reasoning. Engram Adapter improves in-domain accuracy while preserving 99.4%--100.1% of average OOD performance; on LegalBench it slightly exceeds the frozen base model on average, whereas comparable always-on baselines degrade sharply. Mechanistic analyses show that although OOD activations are non-zero, gate and projection attenuation reduce residuals to approximately 0.08% of hidden-state norm, yielding small KL drift and negligible accuracy change. These results suggest conditional activation is a promising route toward modular, retention-preserving domain specialization over frozen backbones.

Problem

• Large language models
• Domain specialization
• Out-of-domain performance
• Parameter-efficient fine-tuning

Innovation

• Conditional Memory
• Retention-Preserving
• Domain Specialization
• Engram Adapter