Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings

2025-06-17

AI Summary

This work addresses the problem of achieving fully reversible, sentence-level text reconstruction from embeddings without modifying large language model (LLM) weights. We propose a zero-shot memory token injection method: a single learnable memory token embedding is optimized, while the LLM’s sequence encoding and prompt-guided decoding pathways remain frozen; the original input text is then reconstructed exactly from the memory token. We empirically demonstrate—for the first time—that mainstream open-weight LLMs (e.g., Llama 3.1-8B) possess inherent full-sequence reversible encoding capability without fine-tuning, achieving 100% token-level reconstruction on English and Spanish benchmarks. The method is cross-lingual, scales across models from 100M to 8B parameters, and supports sequences up to ~240 tokens. This paradigm opens new avenues for lossless text compression, efficient memory-augmented retrieval, and external knowledge storage decoupled from model weights.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In this work, we observe an interesting phenomenon: it is possible to generate reversible sentence embeddings that allow an LLM to reconstruct the original text exactly, without modifying the model's weights. This is achieved by introducing a special memory token, whose embedding is optimized through training on a fixed sequence. When prompted with this embedding, the model reconstructs the fixed sequence exactly. We evaluate this phenomenon across English and Spanish datasets, sequences of up to approximately 240 tokens, and model scales ranging from 100M to 8B parameters. Notably, Llama 3.1 8B successfully reconstructs all tested sequences. Our findings highlight an interesting capability of LLMs and suggest potential applications in memory-based retrieval, compression, and controlled text generation.

Problem

• Generate reversible sentence embeddings without weight modification
• Enable exact text reconstruction using memory tokens
• Evaluate performance across languages and model scales

Innovation

• Reversible embeddings via memory tokens
• Training fixed sequences for exact reconstruction
• Scalable across languages and model sizes