Thought-Retriever: Don't Just Retrieve Raw Data, Retrieve Thoughts for Memory-Augmented Agentic Systems

2026-04-13

AI Summary

This work addresses the limitations of large language models (LLMs) in efficiently leveraging external knowledge due to constrained context lengths and conventional retrieval mechanisms. The authors propose Thought-Retriever, a model-agnostic algorithm that shifts the retrieval target from raw data to the model’s own intermediate reasoning traces—referred to as “thoughts.” By constructing, filtering, and retrieving an evolvable long-term memory of such thoughts, the method overcomes context length constraints and enables agents to continuously self-improve. Integrating thought memory management with LLM reasoning capabilities, the approach is evaluated on a newly introduced benchmark, AcademicEval, designed for ultra-long-context comprehension. Experiments demonstrate an average F1 score improvement of 7.6% and a 16% increase in win rate across AcademicEval and two public datasets, confirming the efficacy of deep reasoning in solving abstract problems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) have transformed AI research thanks to their powerful internal capabilities and knowledge. However, existing LLMs still fail to effectively incorporate the massive external knowledge when interacting with the world. Although retrieval-augmented LLMs are proposed to mitigate the issue, they are still fundamentally constrained by the context length of LLMs, as they can only retrieve top-K raw data chunks from the external knowledge base which often consists of millions of data chunks. Here we propose Thought-Retriever, a novel model-agnostic algorithm that helps LLMs generate output conditioned on arbitrarily long external data, without being constrained by the context length or number of retrieved data chunks. Our key insight is to let an LLM fully leverage its intermediate responses generated when solving past user queries (thoughts), filtering meaningless and redundant thoughts, organizing them in thought memory, and retrieving the relevant thoughts when addressing new queries. This effectively equips LLM-based agents with a self-evolving long-term memory that grows more capable through continuous interaction. Besides algorithmic innovation, we further meticulously prepare a novel benchmark, AcademicEval, which requires an LLM to faithfully leverage ultra-long context to answer queries based on real-world academic papers. Extensive experiments on AcademicEval and two other public datasets validate that Thought-Retriever remarkably outperforms state-of-the-art baselines, achieving an average increase of at least 7.6% in F1 score and 16% in win rate across various tasks. More importantly, we further demonstrate two exciting findings: (1) Thought-Retriever can indeed help LLM self-evolve after solving more user queries; (2) Thought-Retriever learns to leverage deeper thoughts to answer more abstract user queries.

Problem

• retrieval-augmented generation
• context length limitation
• external knowledge integration
• long-term memory
• large language models

Innovation

• Thought-Retriever
• memory-augmented reasoning
• long-term memory
• retrieval-augmented generation
• self-evolving agents