GenKnowSub: Improving Modularity and Reusability of LLMs through General Knowledge Subtraction

2025-05-16

AI Summary

Weak zero-shot generalization in large language models stems from the deep entanglement of general-purpose knowledge and task-specific adaptability. This paper proposes General Knowledge Subtraction (GenKnowSub), a novel framework that decouples these components by constructing two parallel LoRA banks—general-domain LoRAs and task-specific LoRAs—and extracting pure task-residual modules via subtraction. To enable plug-and-play modular composition without fine-tuning, we introduce Arrow, a training-free dynamic routing algorithm. Furthermore, multilingual knowledge distillation is incorporated to enhance cross-lingual transferability. Experiments on Phi-3 and Phi-2 demonstrate an average 3.2% improvement in zero-shot performance and a 47% increase in module reuse rate. The code and datasets are publicly available.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models often struggle with zero-shot generalization, and several modular approaches have been proposed to address this challenge. Yet, we hypothesize that a key limitation remains: the entanglement of general knowledge and task-specific adaptations. To overcome this, we propose a modular framework that disentangles these components by constructing a library of task-specific LoRA modules alongside a general-domain LoRA. By subtracting this general knowledge component from each task-specific module, we obtain residual modules that focus more exclusively on task-relevant information, a method we call general knowledge subtraction (GenKnowSub). Leveraging the refined task-specific modules and the Arrow routing algorithm citep{ostapenko2024towards}, we dynamically select and combine modules for new inputs without additional training. Our studies on the Phi-3 model and standard Arrow as baselines reveal that using general knowledge LoRAs derived from diverse languages, including English, French, and German, yields consistent performance gains in both monolingual and cross-lingual settings across a wide set of benchmarks. Further experiments on Phi-2 demonstrate how GenKnowSub generalizes to weaker LLMs. The complete code and data are available at https://github.com/saharsamr/Modular-LLM.

Problem

• Disentangling general knowledge and task-specific adaptations in LLMs
• Improving zero-shot generalization via modular knowledge subtraction
• Dynamic module selection for cross-lingual performance without retraining

Innovation

• Modular framework disentangles general and task-specific knowledge
• General knowledge subtraction refines task-specific LoRA modules
• Dynamic module selection via Arrow routing algorithm