Output Vector Editing for Memorization Mitigation in Large Language Models

2026-06-17

AI Summary

Large language models are prone to memorizing and reproducing training data, raising significant privacy, copyright, and security concerns. This work proposes a novel constrained optimization–based output vector editing paradigm that precisely identifies key neurons in the MLP layers responsible for memorization and applies minimal perturbations to their output vectors—while preserving their activations—to inject lexical-space interference that redirects their residual stream contributions. Moving beyond conventional approaches that merely zero out activations, the method integrates attention head ablation with multi-modal editing strategies, substantially enhancing memorization suppression. Evaluated on OLMo-7B, the approach identifies 6,831 memorized sequences, achieving an 81.5% suppression rate with single-mode editing and 96.5% with ensemble-mode editing—2.7× higher than zero-ablation baselines—with consistent performance gains scaling with model size.

Abstract

Large language models memorize and reproduce sequences from their training data, creating privacy, copyright, and security risks. Existing neuron-level mitigation methods equate editing with zeroing out neuron activations, but the activation only controls whether a neuron engages; the output vector is what writes to the residual stream and, through superposition, encodes multiple features. We propose output vector editing, a constrained-optimization weight edit that locates a small set of MLP neurons responsible for a memorized continuation and minimally modifies their output vectors to introduce a distractor in vocabulary space, redirecting their residual-stream contributions while leaving activations unchanged. Evaluating on four models from 360M to 7B parameters (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B), we center on OLMo-7B (whose open weights and pretraining corpus enable systematic mining) and mine 6831 memorized sequences, achieving up to 87.9% suppression. The 2.7$\times$ gap over zero ablation on the same located neurons shows the suppression comes from the output-vector edit, not localization alone. Four edit modes span a spectrum from aggressive suppression to minimal redirection; in ensemble they cover 96.5% of memorized sequences, while our recommended single-mode configuration reaches 81.5% with no catastrophic locality failures. We further identify a mechanistic boundary at ${\sim}14%$ of sequences unreachable by MLP-only editing; while these failures are not attention-driven overall, ablating the top contributing attention heads recovers 60--64% of them, with stronger recovery on continuations that copy tokens from the prefix, positioning attention as a complementary fallback rather than a primary mechanism. Edit mode ordering and the success-locality trade-off transfer across all four models, with success rates scaling with model size rather than family.

Problem

• memorization
• large language models
• privacy
• copyright
• security

Innovation

• output vector editing
• memorization mitigation
• constrained optimization
• residual stream redirection
• MLP neuron editing