MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluation

2026-06-29

AI Summary

Current evaluations of agent memory systems often struggle to isolate the impact of memory methods due to confounding changes in language models, embedding models, or retrieval pipelines. This work proposes MemDelta—a controlled memory evaluation protocol that systematically decouples these factors by varying only a single component at a time on the LongMemEval-S benchmark. The study reveals for the first time that swapping embedding models can substantially alter evaluation outcomes, and demonstrates significant differences in how model families (GPT, Gemini, Sonnet) respond to memory strategies. It further shows that self-memory consistently underperforms basic retrieval, and that RAG versus full-context performance rankings reverse across models. Notably, Mem0 matches cloud-based RAG on specific tasks but incurs 50× higher cost. The authors recommend fixing the embedding model and reporting computational costs in future evaluations.

Abstract

Agent memory systems are increasingly evaluated against RAG and full-context baselines, but reported gains often mix changes in the memory method with changes in the language model, embedding model, or retrieval pipeline, making it unclear what is actually being measured. We present MemDelta, a controlled evaluation protocol that varies one component at a time on LongMemEval-S (500 questions, 50+ sessions, three model families). Four findings emerge: (1) verbatim RAG matches full-context GPT-4o-mini (47.2% vs. 49.8%, p = 0.34), but the ranking reverses across models: Gemini gains +14pp from full context, while Sonnet gains +31pp from RAG, partly because it refuses 63% of full-context queries; (2) swapping only the embedding model in an identical pipeline shifts accuracy by +6.2pp at n = 500 (p = 0.004), and Mem0 beats MiniLM-RAG by +11pp but loses to cloud-RAG by 1.2pp, so one variable flips the conclusion; (3) agent self-memory (42%) underperforms basic retrieval (47%); (4) on 2 of 6 question types (n = 88), Mem0 matches cloud RAG (72.7% vs. 73.9%, p = 1.0) at 50x the cost, suggesting narrow rather than general gains. We recommend memory evaluations fix embedding models across comparisons, stratify by model family, and report write-path cost before attributing gains to architecture.

Problem

• agent memory evaluation
• controlled baselines
• hidden confounds
• embedding model variability
• memory system comparison

Innovation

• controlled evaluation
• agent memory
• embedding model
• RAG
• confounding variables