Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents

2026-08-14

AI Summary

This study addresses the lack of guidance and evaluation frameworks for selecting memory substrates in long-horizon agents by establishing a unified benchmark that systematically evaluates seven substrate types across question-answering and decision-making tasks. The research reveals, for the first time, the non-monotonic nature and scenario-dependency of memory performance, demonstrating that no single substrate is universally optimal and that excessive retrieval can impair decision-making. Consequently, this work proposes a novel adaptive memory routing paradigm conditioned on operational contexts. These findings provide critical empirical evidence and methodological support for optimizing long-term memory system design, offering actionable insights into context-aware substrate selection rather than one-size-fits-all approaches.

Abstract

Memory is becoming core infrastructure for long-horizon LLM agents, yet existing evaluations offer limited guidance on which memory substrate, namely the underlying medium in which memory is represented and stored, should be used under different operating regimes. We present a controlled harness evaluation of memory substrates for memory-augmented agents, covering dense and sparse indices, text records, structural stores, hierarchical stores, refinement-based memories, parametric updates, and activation-compatible context mechanisms. Across three backbone models and four benchmark suites spanning user-centric question answering and agent-centric decision-making, we instrument 26 performance and efficiency metrics under a unified harness. Our results show that no single substrate consistently dominates: broad retrieval benefits long-context factual QA, while excessive retrieval can harm sequential decision-making by shifting attention away from action-critical context. Scalability introduces a further routing axis, as substrates that perform well at moderate history lengths can become costly or brittle at longer horizons. These findings motivate substrate routing as a necessary component of adaptive agent memory systems and provide empirical guidance for designing efficient, reliable, and regime-aware long-term memory for LLM agents. Code will be made available upon acceptance.

Problem

• Memory Substrates
• LLM Agents
• Long-horizon Tasks
• Memory Evaluation
• Substrate Routing

Innovation

• Memory Substrates
• Holistic Evaluation
• Substrate Routing
• Memory-Augmented Agents
• Long-horizon LLM