When Your Agent Opens the Chat App: Agent-Controlled Search over Raw Chat Logs Rivals Structured Memory

2026-08-13

AI Summary

This work challenges the necessity of complex memory structures by introducing ReFind, a system that efficiently answers complex questions using only raw chat logs—without constructing summaries, knowledge graphs, or other structured memory representations. ReFind leverages chat-native strategies, including agent-controlled keyword search, conversation-aware ranking fusion, local context expansion, temporal scope filtering, and skipping of already-retrieved conversations, while delegating answer generation to a separate reasoning module, thereby entirely excluding LLMs from the indexing process. Experimental results demonstrate that ReFind achieves an average accuracy of 58.2 on MemoryAgentBench with GPT-4o-mini as the backbone, outperforming the strongest graph- or tree-based systems. On LongMemEval-S and LongMemEval-M, it attains scores of 93.2±3.3 and 89.3±6.0 respectively with GPT-5-mini, showing that lexically driven retrieval, when guided by intelligent agents, can match or even surpass structured memory approaches.

Abstract

Agent-memory systems increasingly buy retrieval quality with structure, transforming raw conversation histories into summaries, embeddings, trees, or knowledge graphs before any question is asked. We ask how much of that benefit comes from the structure itself, rather than from competent retrieval over the raw history. We present ReFind, an agent-controlled search interface that builds no semantic structure at all: it leaves the conversation archive unmodified, indexes it lexically at turn granularity, and combines a generic iterative keyword-search loop with four chat-native controls grounded in empirical refinding work: session-aware rank fusion, local context expansion, temporal narrowing, and skipping already-inspected sessions. A separate reasoning stage answers from the collected evidence. Across a broad suite of conversational-memory tasks (single- and multi-hop QA, event ordering, and fact consolidation), roughly 2,800 questions on precise-retrieval and fact-tracking capabilities evaluated under the incremental multi-turn setting of MemoryAgentBench, ReFind attains the highest mean accuracy (58.2) of any system compared, above the strongest graph- and tree-based memory systems (HippoRAG 2, 53.2), all under a GPT-4o-mini backbone matched to every reused baseline. Controlled comparisons to single-shot BM25, a matched generic-agentic BM25 control, component removals, and agentic dense/hybrid variants separately support the roles of agent control, chat-native controls, and lexical retrieval. On LongMemEval-S/M, the same interface reaches 93.2 +/- 3.3 and 89.3 +/- 6.0 with GPT-5-mini. The results indicate that for precise, evidence-grounded questions over chat archives, much of the benefit credited to elaborate memory structures is recoverable by giving an agent controllable search over the unmodified record, with no LLM-based index construction at all.

Problem

• agent-controlled search
• raw chat logs
• structured memory
• conversational memory
• lexical retrieval

Innovation

• agent-controlled search
• raw chat logs
• lexical retrieval
• chat-native controls
• structured memory