Representation Interventions Enable Lifelong Unstructured Knowledge Control

2025-11-25

AI Summary

To address the challenges of knowledge updating in large language models (LLMs)—namely, difficulty in updating factual knowledge and interference among multiple edits—this paper proposes RILKE, a representation-space intervention method that enables continual, precise, and coexisting updates of unstructured knowledge while keeping the base model weights frozen. Its core innovations are: (1) a localized editing module with synonym robustness to preserve semantic consistency during edits; and (2) a query-adaptive routing mechanism for fine-grained, scalable lifelong knowledge management. Extensive experiments on LLaMA and Qwen demonstrate that RILKE significantly improves edit success rates and generalization across diverse queries, maintains the original model’s general-purpose capabilities, incurs only moderate memory overhead, and scales effectively to large-scale, sustainable knowledge evolution.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) often produce incorrect or outdated content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge. This problem is especially hard for complex, unstructured knowledge in a lifelong setting, where many edits must coexist without interference. We introduce RILKE (Representation Intervention for Lifelong KnowledgE Control), a robust and scalable method that treats knowledge control as interventions within the model's representation space. Leveraging representation-space expressiveness, we identify two properties enabling RILKE to deliver fine-grained control over complex, unstructured knowledge while maintaining general utility with frozen base weights. During training, RILKE learns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference. In inference, a query-adaptive router selects the appropriate module to guide the model's generation. In evaluation on knowledge editing benchmarks with LLaMA and Qwen models, RILKE is scalable to large-scale datasets, demonstrating high edit success, strong paraphrase generalization, and preserving general utility with modest memory overhead. These results show RILKE is an effective and scalable solution for lifelong knowledge control in LLMs.

Problem

• Updating LLM knowledge without costly retraining
• Managing complex unstructured knowledge in lifelong settings
• Preventing interference between multiple coexisting knowledge edits

Innovation

• Representation-space interventions for knowledge control
• Learns paraphrase-robust localized modules in subspace
• Query-adaptive router selects modules during inference