EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory

2026-06-19

AI Summary

Traditional embedding models, due to their static treatment of text, struggle to capture contextual dynamics and temporal information, thereby limiting long-context retrieval and agent memory capabilities. This work proposes EvoEmbedding, the first framework to introduce an evolvable embedding mechanism that jointly generates context-aware dynamic representations by maintaining a continuously updated latent memory during sequential processing. The approach incorporates a memory queue to prevent representational collapse and employs segmented batch processing to accelerate training, enabling end-to-end joint optimization on a newly curated EvoTrain-180K dataset. Experiments demonstrate that EvoEmbedding outperforms larger, specialized models on multiple long-context retrieval benchmarks, supports inference lengths up to ten times beyond its training sequence length, significantly enhances agent workflow performance, and generalizes effectively to downstream tasks such as personalization.

Abstract

Existing embedding models are inherently static: they encode text segments in isolation, ignoring their surrounding context and temporal order. This paper introduces EvoEmbedding, a novel embedding model that generates evolvable representations for retrieval. It is tailored for long-context scenarios, where information is dynamic, sequential, and requires continuous state tracking. Our design is simple: EvoEmbedding maintains a continuously updated latent memory as it sequentially processes inputs, and uses it alongside the raw content to jointly generate evolvable embeddings. Consequently, for the same query, our model adapts its representation to retrieve distinct targets based on the evolving context, going beyond static semantic search. To equip the model with this capability, we construct EvoTrain-180K, a diverse dataset for the joint optimization of latent memory and retrieval. Furthermore, we introduce a memory queue to prevent representation collapse during recurrent encoding, alongside segment-batching techniques that tackle significant length variance and accelerate training by 3.8$\times$. Extensive experiments show that our model not only outperforms larger-scale specialists (e.g., Qwen3-Embedding-8B and KaLM-Embedding-Gemma3-12B) across a range of long-context retrieval benchmarks, but also generalizes well to downstream tasks (e.g., personalization) with contexts 10$\times$ longer than its training window. Notably, EvoEmbedding seamlessly integrates into agentic workflows to boost performance. For instance, a naive RAG pipeline equipped with our model surpasses dedicated agentic memory systems. Project Page: https://clare-nie.github.io/EvoEmbedding.

Problem

• long-context retrieval
• evolvable representations
• agentic memory
• static embedding
• temporal context

Innovation

• evolvable embeddings
• long-context retrieval
• latent memory
• agentic memory
• segment-batching