Fixed RAG Compression Collapses Measured Reader Scaling

2026-06-19

AI Summary

This work challenges the prevailing assumption in retrieval-augmented generation (RAG) systems that fixed compression is reader-agnostic, demonstrating instead that it obscures performance gains and distorts model rankings. Through nine out of ten experiments showing statistically significant negative correlations (p < 0.05) between compression-induced gains and reader baseline capabilities, the study refutes the neutrality of compression. The authors introduce a comprehensive evaluation framework encompassing structured compilation, generic summarization, and query-focused summarization, analyzing 177,000 compressed samples from three trainable compressors. Results reveal that generic summarization induces ranking reversals in 31% of models and that fixed compression masks up to 80% of reader improvement. To support reproducible research, the team releases ragscale, a toolkit enabling efficient auditing and standardized evaluation.

Abstract

Retrieval-Augmented Generation (RAG) compression papers often evaluate a compressor on one to three readers and treat the compressed evidence layer as evaluation-neutral. We show this assumption is false: fixed compression can raise average accuracy while hiding reader upgrades and reversing model rankings. Across 20 readers and ten domain-method settings over four QA benchmarks and one summarization benchmark, compression gain decreases with reader baseline (nine of ten settings significant, p < 0.05). Generic summarization flips 31% of pairwise model rankings on LongMemEval-S, and a fixed HotpotQA compressor hides 80% of the raw upgrade from Qwen 7B to GPT-4.1-mini. Two opposing forces explain this paradox: compression rescues weak readers by removing noise they cannot filter, and harms strong readers by dropping details they would have used. The pattern appears across structured compilation, generic summarization, three trained compressor families, query-focused summarization, and an external audit of nine published compression papers. We release ragscale, a toolkit built on 177,000 row-level compression transitions, so any compression paper can audit reader scaling with three readers in one day.

Problem

• Retrieval-Augmented Generation
• RAG compression
• reader scaling
• model evaluation
• compression bias

Innovation

• RAG compression
• reader scaling
• model ranking reversal
• compression bias
• ragscale