AgentIR: A Workload-Adaptive Cascade Retrieval Substrate for Long-Term Conversational Memory

2026-05-24

AI Summary

Traditional information retrieval systems struggle to meet the demands of long-context conversational memory scenarios, which involve dynamically growing indexes, evolving query types, and sub-10-millisecond latency constraints. This work proposes a workload-adaptive cascaded retrieval framework featuring a novel zero-training, BM25-confidence-based self-tuning routing mechanism that dynamically bypasses costly dense retrieval paths. The approach integrates reciprocal rank fusion (RRF) with intelligent re-ranking strategies and leverages time-partitioned inverted indexes, chunked index construction, and GPU acceleration to significantly enhance efficiency. On LongMemEval, it skips 63% of dense queries without sacrificing accuracy, achieving 2.67–5.76× speedups; on LoCoMo, it automatically omits all dense computations, yielding a 9× throughput gain and scaling the index size by 1,234× with only a 3.6× latency increase.

Abstract

Long-term conversational memory is a retrieval workload classical IR was not built for: the index grows during the query stream, query types shift intra-session, and the latency budget per retrieval is sub-10 ms. Lucene-class engines treat the index as static and the query as stateless, leaving the workload's structure unexploited. AgentIR treats fusion as a per-query decision along two axes: which fusion to apply (BM25, Dense, RRF, or agent-aware RRF), and whether the ~52 ms dense channel is worth running at all. The second axis is a confidence-triggered cascade router that decides from the BM25 top-k margin alone and re-tunes across workloads without retraining. On LongMemEval (n=500), where the dense channel does add information, the cascade skips 63% of queries at parity LLM-judged accuracy (2.67x faster under two judges, paired bootstrap p>=0.88); per-qtype thresholds extend this to 5.76x under 5-fold cross-validation. On LoCoMo (n=1,982), where BM25 alone is already the strongest single system, the same trigger auto-tunes to a 100% skip rate (132x faster, +0.089 Hit@5). Capacity on a shared 8-core VM rises from ~154 to ~1,400 concurrent agents (9x). Underneath the cascade, a time-partitioned index does O(log 1/epsilon) work independent of corpus size: 1234x corpus growth costs only 3.6x latency, ending in 1769x over sequential at sub-100 us p50 on 5M records. At parity quality with Lucene on 9 BEIR datasets up to 8.8M docs, the substrate runs 10x geo-mean over Pyserini 8T and 11x over PISA-1T BlockMax-WAND; an A100 reaches 1.8-39x over Pyserini 8T; chunked index build sustains 56.8K docs/sec on MS MARCO. Three subtle BM25/GPU correctness pitfalls that silently regress nDCG@10 by 6-8x are documented and fixed; post-fix CPU and GPU agree within 0.0002 nDCG@10 on all eight datasets that fit a single A100.

Problem

• long-term conversational memory
• workload-adaptive retrieval
• cascade retrieval
• low-latency retrieval
• dynamic index

Innovation

• cascade retrieval
• workload-adaptive routing
• time-partitioned index
• long-term conversational memory
• BM25/GPU correctness