Co-LMLM: Continuous-Query Limited Memory Language Models

2026-07-08

AI Summary

Traditional large language models struggle to dynamically update and trace their internalized knowledge. This work proposes a novel limited-memory language model that retrieves and injects verifiable factual information from an external knowledge base on demand. The approach introduces a continuous vector query mechanism, eliminating reliance on structured relational databases, and enables automatic annotation and retrieval of free-form factual snippets from arbitrary text. It further integrates a continuous key-value knowledge store with knowledge-enhanced language modeling. Experiments demonstrate consistent superiority over existing limited-memory language models and standard LLMs across multiple model scales. Notably, at 360M parameters, the method achieves factual accuracy comparable to GPT-4o-mini while attaining lower perplexity with substantially less pretraining data.

Abstract

Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights. During generation, the model then fetches knowledge from the KB as needed. This recently introduced paradigm provides multiple advantages, including knowledge control capabilities that remain beyond conventional LLMs. We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries. CO-LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation. We pair this design with an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior work's restriction to Wikipedia. Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pretrained on 40x more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.

Problem

• limited memory language models
• knowledge externalization
• factual knowledge control
• retrieval-augmented generation
• continuous queries

Innovation

• continuous-query
• limited memory language model
• knowledge retrieval
• fact grounding
• vector-based knowledge base