memory systems

Large language models (LLMs) excel at complex contextual understanding but exhibit pronounced capability asymmetry—struggling to stably generate long, equally sophisticated texts. Method: We systematically establish a unified “context engineering” framework, proposing a four-dimensional taxonomy encompassing retrieval, generation, processing, and management. Based on a systematic review and architectural analysis of 1,300+ papers, we construct the first comprehensive context engineering technology map; identify the intrinsic mechanisms underlying the understanding–generation capability mismatch; and delineate architectural integration pathways for four key application paradigms: retrieval-augmented generation, memory modeling, tool integration, and multi-agent coordination. Contribution/Results: The work delivers a standardized conceptual framework, a strategic technology roadmap, and identified critical breakthrough directions—providing both theoretical foundations and practical guidance for developing advanced context-aware AI systems.

To address GPU memory overflow and I/O bottlenecks caused by KV caching in long-context LLM serving, this paper proposes Strata, a hierarchical caching framework. Methodologically, Strata introduces (1) GPU-accelerated I/O via CUDA kernels that aggregate non-contiguous page reads to mitigate cache fragmentation; (2) decoupled CPU–GPU memory layouts to enable efficient coordination across heterogeneous storage tiers; and (3) a cache-aware dynamic request scheduler that explicitly models load latency and overlaps I/O with computation. Implemented atop SGLang, Strata achieves up to 5× lower first-token latency compared to vLLM+LMCache and 3.75× higher throughput for long-context workloads versus TensorRT-LLM—while preserving short-context performance.

This work addresses the escalating inference costs in long-horizon LLM agent execution caused by accumulating context, where existing pruning or memory eviction strategies often disrupt prompt cache continuity, leading to prefix mismatches and cache invalidation. To tackle this, the authors propose TokenPilot, a dual-granularity context management framework that globally employs ingestion-aware compression to stabilize the prompt prefix and filter environmental noise, while locally applying a lifetime-aware eviction mechanism to conservatively offload context once its task relevance diminishes. TokenPilot is the first approach to achieve efficient context compression without compromising cache continuity, effectively balancing textual sparsity and cache efficacy. Experiments on PinchBench and Claw-Eval demonstrate that TokenPilot reduces inference costs by up to 61%/56% in isolated mode and 61%/87% in continuous mode, with no degradation in task performance, and has been integrated into LightMem2.

To address the challenges of limited computational resources and bandwidth for large language model (LLM) services in mobile edge computing, as well as high retrieval latency and substantial update overhead induced by conventional RAG-based cache management, this paper proposes the first deep reinforcement learning (DRL)-based adaptive context caching framework. The framework jointly models user intent, semantic similarity, and cache-miss cost to enable dynamic, optimal caching decisions. It integrates semantic-aware caching, seamless RAG incorporation, and context-aware cache replacement. Experimental results demonstrate that the framework achieves over 80% cache hit rate within only 11 training episodes, reduces retrieval latency by 40%, and cuts local cache update overhead by 55%. These improvements significantly enhance the real-time performance and scalability of edge-deployed LLM services.

Existing MLLM services employ context caching that only reuses fixed-prefix KV states, rendering them ill-suited for text-image interleaved inputs and dynamic prefix variations in multimodal RAG—leading to substantial redundant recomputation. This work proposes a **position-agnostic multimodal KV caching mechanism**, the first to decouple positional encoding from cached KV states, enabling flexible, cross-sequence and cross-modal KV reuse. Our approach integrates on-demand recomputation, multimodal token alignment, dynamic scheduling, and disk-persistent loading. Experiments demonstrate that, while preserving near-lossless accuracy (<0.3% degradation), our method reduces end-to-end latency by up to 54%, significantly outperforming state-of-the-art caching schemes.

In hybrid large language models (LLMs) combining attention and recurrent layers, prefix caching suffers from severe inefficiency due to in-place state updates in recurrent layers—rendering conventional caches effective only under exact sequence matches and yielding extremely low reuse rates. Method: This paper introduces the first prefix caching system tailored for hybrid LLMs. It innovatively integrates reuse probability prediction with a dynamic admission and eviction policy guided by the computational savings-to-memory overhead ratio. The design further incorporates cache-hotness modeling, classification of sequence reuse patterns, and lightweight state snapshot management. Results: Evaluated across diverse hybrid models and realistic workloads, the system achieves up to a 34.4× improvement in token-level cache hit rate (reaching 71.1%) and reduces first-token generation latency by 617 ms. These gains significantly enhance caching efficiency and inference throughput.

This study addresses the substantial memory pressure and performance trade-offs in large language model (LLM) inference caused by KV cache growth with increasing context length and concurrent requests. The authors systematically evaluate three state-of-the-art KV cache management frameworks—vLLM, InfiniGen, and H2O—across diverse workloads, model scales, and sparsity conditions, measuring their latency, throughput, and memory efficiency. By integrating key techniques such as tensor offloading, token eviction heuristics, and speculative scheduling, the work provides the first comprehensive characterization of the operational boundaries of these strategies under varied deployment scenarios. It identifies optimal configurations under specific memory and performance constraints, offering empirical insights and practical guidance for designing efficient LLM inference systems.

This work addresses the limitations of traditional retrieval-augmented generation (RAG), which treats memory as a static, stateless, read-only lookup table and thus struggles to support knowledge evolution and state updates in long-term interactions. To overcome this, the authors propose the Continuum Memory Architecture (CMA), the first formalized memory system that enables stateful evolution through persistent storage, selective retention, temporal chaining, associative routing, and integration of high-level abstractions. CMA facilitates dynamic memory accumulation, updating, and disambiguation over time. Experimental results demonstrate that CMA significantly outperforms conventional RAG on tasks involving knowledge updating, temporal reasoning, associative recall, and contextual disambiguation. This advancement equips long-horizon large language model agents with essential memory capabilities, while also uncovering new challenges related to latency, semantic drift, and interpretability.

To address the limited context capacity and inefficient repeated-prompt inference in small-parameter language models (LLMs), this paper proposes a model-agnostic KV cache reuse mechanism. The method leverages Sentence-BERT to generate sentence embeddings and enables lossless cross-prompt KV state reuse when a new prompt shares an exact prefix with a historical one—achieved via embedding-based retrieval and serialized cache reconstruction. It supports CPU-side cache persistence and hot reloading, ensuring both generality and deployment friendliness. Experiments demonstrate significantly reduced inference latency under prefix-overlap conditions, with negligible degradation in semantic fidelity; when no overlap exists, behavior matches the baseline exactly, confirming robustness and practicality. To our knowledge, this is the first work achieving lossless, cross-prompt KV cache reuse while maintaining full compatibility with the standard Hugging Face Transformers framework.

This work addresses the limitations of large language models in long-context reasoning, which stem from the quadratic complexity of attention mechanisms and the high memory overhead of key-value (KV) caching. To mitigate these issues, the authors propose a boundary-aware chunking strategy that preserves local semantic coherence and introduce a recursive hierarchical index grounded in the triangle inequality. This index transforms KV cache retrieval from linear scanning into a logarithmic-time pruning process. Coupled with a lazy-update mechanism, the approach enables efficient streaming generation. Evaluated across multiple benchmarks, the method achieves up to 3.6× end-to-end inference speedup with negligible degradation in model performance, significantly outperforming existing techniques such as Quest and ClusterKV.

This work addresses the memory and efficiency bottlenecks in large language models caused by the linear growth of KV cache during long-context reasoning. Existing fixed Soft Token compression methods suffer from a lack of input adaptivity and incur irreversible information loss. To overcome these limitations, the authors propose Meta-Soft, a novel framework that introduces a composable meta-token mechanism. It constructs a learnable orthogonal basis as a meta-bank and employs a Gumbel-Softmax selector to dynamically synthesize k input-aware Soft Tokens for probing salient information. Coupled with an attention-flow-driven semantic redistribution strategy, the framework reallocates information from pruned tokens to retained ones. This approach enables context-aware KV cache compression, achieving state-of-the-art performance across multiple benchmarks while preserving both computational efficiency and semantic fidelity.