Extractable Memorization From First Principles

2026-07-14

AI Summary

This work addresses the tendency of existing methods for extracting memorized content from large language models to overestimate model capabilities or misclassify generated text as memorized. The authors propose a calibration framework based on matched comparisons, which evaluates the generation probability of a training sequence against that of similar non-training sequences. By integrating conformal testing (for population-level analysis) and exhaustive enumeration (for single-document assessment), the framework rigorously defines extractable memorization as requiring both statistically significant evidence of memory and high-generation probability within realistic sampling budgets. Experiments on models such as OLMo 2 32B and Llama 3.1 70B reveal that prior methods exhibit false positive rates as high as 24%, whereas the proposed approach calibrates an effective memorization threshold as low as 1e-27, substantially improving the reliability of memory detection.

Abstract

Recent work on extractable memorization in LLMs suffers from two contrasting validity problems. Some studies overstate extraction, e.g., relying on sequences too short to distinguish memorization from predictability. Others imply that extraction is unreliable evidence of memorization, since models can also reproduce real-world text they weren't explicitly trained on. In different ways, both overlook what makes a valid extraction claim: the model must generate a training sequence with high enough probability to indicate memorization. To determine what's high enough, one has to perform a matched comparison: measuring the generation probabilities of both the training sequences of interest and comparable non-training sequences. Because non-training sequences cannot have been memorized, their probabilities provide a baseline for predictability; a training sequence exceeding this baseline provides evidence of memorization. We formalize matched comparisons in two ways: (1) a conformal test that calibrates a threshold to a chosen FPR when training and non-training sequences are sampled from populations, and (2) a census that calibrates against a matched non-training document when the object is a single document (e.g., a book). We show that matched comparisons enable rigorous, calibrated memorization claims, and reveal where prior setups have validity issues. For instance, on Wikipedia OLMo 2 32B reproduces non-training 10-token suffixes roughly 24% as often as training ones: that share of the training generation rate reflects false positives, not memorization. For Llama 3.1 70B on books, the thresholds we calibrate are as low as 1e-27, supporting memorization claims for sequences that no feasible sampling budget would extract. Based on these results, we refine "extractable memorization" to require a valid memorization claim and near-certain generation within a realistic budget.

Problem

• extractable memorization
• large language models
• validity
• generation probability
• training data

Innovation

• extractable memorization
• matched comparison
• conformal test
• generation probability
• memorization baseline