Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory

2026-03-04

AI Summary

This work addresses the limitation of large language model (LLM) agents in long-horizon tasks, where finite context windows hinder effective utilization of full interaction histories. To overcome this, the authors propose Memex, a mechanism that compresses the working context through structured summarization and stable indexing, while storing raw interactions in an external experience database. Integrated within a reinforcement learning framework—MemexRL—the approach jointly optimizes memory writing, indexing, and retrieval strategies. Evaluated across multiple long-horizon tasks, Memex significantly reduces context overhead without sacrificing critical evidence, leading to substantial improvements in task success rates. The results demonstrate that Memex effectively balances decision quality with computational efficiency, enabling scalable and cost-effective deployment of LLM agents in extended interactive scenarios.

Technology Category

• Multiagent Systems: Mechanism Design
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language model (LLM) agents are fundamentally bottlenecked by finite context windows on long-horizon tasks. As trajectories grow, retaining tool outputs and intermediate reasoning in-context quickly becomes infeasible: the working context becomes prohibitively long, eventually exceeds the context budget, and makes distant evidence harder to use even when it is still present. Existing solutions typically shorten context through truncation or running summaries, but these methods are fundamentally lossy because they compress or discard past evidence itself. We introduce Memex, an indexed experience memory mechanism that instead compresses context without discarding evidence. Memex maintains a compact working context consisting of concise structured summaries and stable indices, while storing full-fidelity underlying interactions in an external experience database under those indices. The agent can then decide when to dereference an index and recover the exact past evidence needed for the current subgoal. We optimize both write and read behaviors with our reinforcement learning framework MemexRL, using reward shaping tailored to indexed memory usage under a context budget, so the agent learns what to summarize, what to archive, how to index it, and when to retrieve it. This yields a substantially less lossy form of long-horizon memory than summary-only approaches. We further provide a theoretical analysis showing the potential of the Memex loop to preserve decision quality with bounded dereferencing while keeping effective in-context computation bounded as history grows. Empirically, on challenging long-horizon tasks, Memex agent trained with MemexRL improves task success while using a significantly smaller working context.

Problem

• long-horizon tasks
• context window limitation
• experience memory
• evidence retention
• LLM agents

Innovation

• indexed memory
• long-horizon agents
• experience database
• reinforcement learning
• context compression