Pull: Lazy Materialization of Working Memory for Stateful LLM Conversations

2026-09-13

AI Summary

为解决长对话中全上下文注入成本高及信息丢失问题，提出Pull方法，通过可逆的按需实例化历史对话轮次来保持会话状态。

Abstract

As LLM conversations grow to hundreds of turns, full-context injection incurs $O(N^2)$ cumulative token costs, while lossy summarization or hard truncation irreversibly discards historical state. We propose Pull, a session router that maintains an addressable metadata directory via a local, deterministic Purifier (zero LLM calls, millisecond-level latency). At query time, the LLM lazily materializes only the turns it needs; unmaterialized turns remain accessible but collapsed. Unlike irreversible compression, Pull's materialization is reversible; subsequent queries can expand any collapsed turn. On LoCoEval (128 conversations, 12,780 turns), Pull reduces per-query context tokens (Phase 2) by 75.1 percent on single-hop tasks with equivalent quality ($Δ= -0.002$, n.s.) and by 72.0 percent on multi-hop tasks with no quality loss ($Δ= +0.017$). A controlled routing benchmark (7,831 queries x 10 methods) shows that entity lifecycle tracking is empirically a prerequisite for distance-independent routing. On BEAM 1M (14 conversations, 263 questions), Pull improves F1 by +55.2 percent over a truncation baseline.

Problem

• LLM Conversations
• Token Costs
• Historical State
• Working Memory

Innovation

• Lazy Materialization
• Addressable Metadata Directory
• Deterministic Purifier
• Reversible Compression
• Session Router