Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents

2026-07-06

AI Summary

Traditional language agents are constrained by high latency and limited reasoning capabilities due to infrequent access to external memory, hindering the sustained availability akin to human working memory. This work proposes embedding memory storage directly within the agent’s reasoning loop, constructing an end-to-end language agent system through in-process vector storage, lightweight local embedding models, and efficient read-write strategies. The study demonstrates for the first time that this architecture reduces retrieval latency to 80–165 microseconds (p50), significantly improving task recall accuracy from 0/5 to 3.6–4.8/5 under a fixed latency budget, while achieving zero data loss across 244 write operations. These results provide a scalable engineering realization supporting the extended mind hypothesis.

Abstract

Language agents run a loop - observe, reason, act - but the memory they reason over sits outside it: a store queried at most once per turn. We study the regime where memory moves inside the loop, read and written on every step. The obstacle has always been latency: networked stores answer in tens to hundreds of milliseconds, and in-loop retrieval can inflate end-to-end latency by up to 83x when retrieval is expensive. Prior work manages that cost rather than questioning it: serving-layer scheduling hides it, "memory-first" designs ration retrieval to once per turn. We argue latency is a property of where the store lives, not the in-loop pattern: an in-process store answers in ~100us, three orders of magnitude below the network regime, and at that speed the per-step tax collapses. By the extended-mind thesis's parity principle, a store fast enough to be constantly and directly available becomes extended working memory, not a tool the agent merely consults. The premise is causal: holding a fixed per-turn memory-latency budget and varying only the store's answer speed, redundant actions rise monotonically with latency - 0.0 of 12 at in-process speed, 7.2 of 12 at a 110ms cloud round trip (gpt-5-nano, gpt-5-mini; exact permutation p=0.0079). We demonstrate the regime end-to-end: across four GPT-5-class models under a bounded window, recall improves from 0/5 to 3.6-4.8/5 with in-loop memory, store ops at p50 80-165us - though an instructed restate-every-reply baseline also solves it perfectly, at a token cost that grows with the working set. The store never lost a fact in any run (244 of 244 writes kept); every miss traces to the agent's read policy, not the store. Our measurements also relocate the bottleneck: the dominant per-step cost is embedding (~200-400ms over the network); pairing the in-process store with a small local embedder returns the complete operation to a measured ~40us.

Problem

• language agents
• in-loop retrieval
• memory latency
• extended working memory
• real-time memory access

Innovation

• in-loop memory
• extended working memory
• ultra-low-latency storage
• language agents
• embedding bottleneck