Decoupled Mixture-of-Experts for Parametric Knowledge Injection

2026-06-12

AI Summary

This work addresses the trade-off between flexibility and integrability when injecting external, domain-specific, or time-sensitive knowledge into large language models—a challenge that often leads to catastrophic forgetting, knowledge conflicts, or high update costs. To overcome these limitations, the authors propose Decoupled Mixture of Experts (DMoE), which encodes external knowledge into independently updatable expert modules. A lightweight, uncertainty-aware router dynamically activates relevant experts only when the base model exhibits insufficient knowledge. Crucially, experts are attached solely to the final feed-forward layer, enabling efficient autoregressive inference and reuse of key-value caches. DMoE is the first approach to fully decouple both experts and the router from the base model, achieving high parameter efficiency in knowledge injection while preserving generation quality. It significantly outperforms retrieval-augmented and adapter-based baselines across multiple knowledge-intensive benchmarks.

Abstract

Knowledge injection aims to equip large language models (LLMs) with external, domain-specific, or time-sensitive knowledge. Existing approaches typically face a trade-off between flexibility and integration: retrieval-augmented generation keeps knowledge outside the model but only provides prompt-level augmentation, whereas post-training based methods encode new knowledge into shared parameters but may introduce catastrophic forgetting, knowledge conflict, and costly updates. In this paper, we propose Decoupled Mixture-of-Experts (DMoE), a modular architecture for parametric knowledge injection that decouples both experts and the router from the base model. DMoE converts external knowledge corpora into independently updatable expert modules and uses a lightweight uncertainty-aware router to activate relevant experts only when the base model lacks sufficient knowledge during generation. To support efficient auto-regressive inference, DMoE attaches experts only to the final-layer feed-forward network, preserving KV-cache reuse while enabling parameter-level knowledge augmentation. Experiments on knowledge-intensive benchmarks show that DMoE consistently improves answer quality over retrieval and adapter-based baselines.

Problem

• knowledge injection
• large language models
• catastrophic forgetting
• retrieval-augmented generation
• parametric knowledge

Innovation

• Decoupled Mixture-of-Experts
• parametric knowledge injection
• uncertainty-aware routing
• modular architecture
• knowledge-intensive generation