Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation

2026-08-31

AI Summary

研究探讨了量化KV缓存对检索增强生成系统忠实性的影响，使用Qwen2.5-7B-Instruct模型在不同量化级别下评估准确性和忠实性。

Abstract

Retrieval-augmented generation systems can precompute and store key-value caches of retrieved documents to avoid re-encoding context at every query. Quantizing these caches further reduces storage, but no prior work asks whether compression damages faithfulness, whether responses remain grounded in the retrieved evidence. Faithfulness and accuracy are not equivalent: a model can produce a correct answer that is no longer supported by the context it was given. We evaluate Qwen2.5-7B-Instruct under INT8 and INT4 quantization on RGB and HotpotQA, measuring both accuracy and faithfulness with a hallucination detector, NLI entailment, and an LLM judge. INT8 is near-lossless across both metrics. INT4 reduces accuracy and, more critically, even among answers that remain factually correct, over 90% of faithfulness changes are negative, i.e., accuracy metrics are blind to this regression. The harm grows under noisy retrieval and with more retrieved chunks. Faithfulness must be audited before compressed caches are deployed.

Problem

• Retrieval-augmented generation
• KV-Cache Quantization
• Faithfulness

Innovation

• Faithfulness
• Quantization
• KV-Cache
• Retrieval-Augmented Generation
• Accuracy