Decoupling Knowledge and Task Subspaces for Composable Parametric Retrieval Augmented Generation

2026-04-29

AI Summary

This work addresses the instability and degraded knowledge focus in parametric retrieval-augmented generation (RAG) when combining multiple documents, caused by the entanglement of task-specific behavior and document-specific knowledge within shared adapters. To resolve this, the authors propose an orthogonal subspace decomposition strategy that explicitly disentangles these two aspects by modeling them as separate task LoRA and document LoRA modules. During training, orthogonality constraints are enforced to ensure these subspaces remain distinct, thereby decoupling task capabilities from factual knowledge. The approach substantially enhances compositional robustness, generation stability, and factual accuracy of parametric RAG in multi-document settings. Extensive experiments across diverse knowledge-intensive tasks and model scales demonstrate the method’s consistent effectiveness and generalizability.

Abstract

Parametric Retrieval-Augmented Generation (PRAG) encodes external documents into lightweight parameter modules that can be retrieved and merged at inference time, offering a promising alternative to in-context retrieval augmentation. Despite its potential, many PRAG implementations train document adapters with task-supervised objectives, which may cause each adapter to encode both document-specific facts and reusable task-solving behavior. This entanglement may make adapter composition less reliable: when multiple adapters are merged at inference time, their overlapping task behaviors can accumulate together with document-specific updates, potentially making the merged adapter less stable and less focused on the intended document knowledge. To examine this issue, we explore Orthogonal Subspace Decomposition (OSD), an adapter-training setup that separates reusable task behavior from document-specific knowledge adapters. Concretely, we first train a Task LoRA to capture reusable task behavior, and then train document LoRAs to encode document-specific knowledge in a orthogonal subspace. This setup provides a controlled way to examine how orthogonalizing task and document LoRA updates affects adapter composition in multi-document PRAG. Experiments across multiple knowledge-intensive tasks and model scales suggest that this orthogonalization strategy can improve compositional robustness in parametric RAG, especially when multiple document adapters are merged.

Problem

• Parametric Retrieval-Augmented Generation
• adapter composition
• task-document entanglement
• compositional robustness
• knowledge-intensive tasks

Innovation

• Parametric Retrieval-Augmented Generation
• Orthogonal Subspace Decomposition
• LoRA
• Adapter Composition
• Knowledge Decoupling