NGM: A Plug-and-Play Training-Free Memory Module for LLMs

2026-05-16

AI Summary

This work addresses the limitations of existing conditional memory modules, which rely on trained memory embeddings and thus require additional training and suffer from limited flexibility. The authors propose the Non-parametric Gated Memory (NGM) module—the first plug-and-play, training-free memory augmentation mechanism. NGM constructs context-aware n-gram representations by averaging pretrained word embeddings through a causal n-gram encoder and dynamically injects retrieved knowledge into language model representations via a non-parametric cosine gating mechanism, eliminating the need for a separate memory table or explicit retrieval pipeline. Evaluated across the Qwen3 model family (0.6B–14B), NGM yields consistent gains of 0.5–1.2 points on average, with notable improvements of 3.0 and 3.03 points on LiveCodeBench and GPQA, respectively, and a 1.53-point gain on the multimodal benchmark MMStar, significantly enhancing both knowledge acquisition and deployment flexibility in large language models.

Abstract

Recent studies introduce conditional memory modules that decouple knowledge storage from neural computation, enabling more direct knowledge access. Compared to MoE, which relies on dynamic computation paths, explicit lookup provides a more efficient knowledge retrieval mechanism. However, these approaches still depend on learned memory embeddings, requiring additional training and limiting flexibility. To address this, we propose N-gram Memory (NGM), a training-free, plug-and-play module composed of a Causal N-Gram Encoder and a Cosine-Gated Memory Injector. The Causal N-Gram Encoder directly averages the pretrained token embeddings of the backbone model to construct N-gram representations, thereby eliminating the need to train separate N-gram embeddings from scratch. This design requires neither an additional memory table nor a retrieval pipeline. The Cosine-Gated Memory Injector then uses a non-parametric cosine gate with ReLU to modulate the retrieved embeddings into the contextual representations. We evaluate NGM on the Qwen3 series from 0.6B to 14B across eight benchmarks. NGM improves average performance by 0.5 to 1.2 points, with particularly clear gains on code generation and knowledge-intensive tasks (e.g., +3.0 on LiveCodeBench and +3.03 on GPQA for Qwen3-14B). Moreover, NGM also improves performance in multimodal benchmarks (e.g., MMStar +1.53 on Qwen3-VL-2B).

Problem

• conditional memory
• training-free
• knowledge retrieval
• plug-and-play
• memory module

Innovation

• training-free
• plug-and-play
• N-gram memory
• cosine-gated injection
• knowledge retrieval