MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning

2026-03-02

AI Summary

This work addresses the challenge of efficient and cost-effective memory retrieval for large language models (LLMs) in long-horizon tasks, where existing approaches struggle to balance retrieval accuracy with computational overhead. The authors propose a lightweight proxy model that leverages a task-outcome-driven reinforcement learning mechanism to optimize long-term memory management without imposing additional burden on the primary LLM. The method uses task success as a reward signal and integrates curriculum learning, model fusion strategies, and a multi-turn interaction evaluation framework. Evaluated across eight LLM memory benchmarks—including Deep Research tasks—the approach matches or exceeds state-of-the-art performance in both retrieval accuracy and downstream task effectiveness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

As Large Language Models (LLMs) are increasingly used for long-duration tasks, maintaining effective long-term memory has become a critical challenge. Current methods often face a trade-off between cost and accuracy. Simple storage methods often fail to retrieve relevant information, while complex indexing methods (such as memory graphs) require heavy computation and can cause information loss. Furthermore, relying on the working LLM to process all memories is computationally expensive and slow. To address these limitations, we propose MemSifter, a novel framework that offloads the memory retrieval process to a small-scale proxy model. Instead of increasing the burden on the primary working LLM, MemSifter uses a smaller model to reason about the task before retrieving the necessary information. This approach requires no heavy computation during the indexing phase and adds minimal overhead during inference. To optimize the proxy model, we introduce a memory-specific Reinforcement Learning (RL) training paradigm. We design a task-outcome-oriented reward based on the working LLM's actual performance in completing the task. The reward measures the actual contribution of retrieved memories by mutiple interactions with the working LLM, and discriminates retrieved rankings by stepped decreasing contributions. Additionally, we employ training techniques such as Curriculum Learning and Model Merging to improve performance. We evaluated MemSifter on eight LLM memory benchmarks, including Deep Research tasks. The results demonstrate that our method meets or exceeds the performance of existing state-of-the-art approaches in both retrieval accuracy and final task completion. MemSifter offers an efficient and scalable solution for long-term LLM memory. We have open-sourced the model weights, code, and training data to support further research.

Problem

• Long-term memory
• Memory retrieval
• Large Language Models
• Cost-accuracy trade-off
• Computational overhead

Innovation

• proxy reasoning
• memory retrieval offloading
• outcome-driven reinforcement learning
• long-term LLM memory
• lightweight retrieval model