Where Should a Document Live: Context, Representations, or Parameters?

2026-09-15

AI Summary

研究比较了基于表示和参数调整方法在知识密集型任务中的效率与性能，发现Cartridges（KV）在多数情况下优于其他方法，但存在灾难性遗忘问题。

Abstract

To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding.

Problem

• large language models
• new information
• efficiency
• cost
• performance

Innovation

• Cartridges (KV)
• parametric methods
• knowledge-intensive benchmarks
• catastrophic forgetting
• multi-document retrieval