Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMs

2026-06-15

AI Summary

This work investigates how an adversary, limited to poisoning only a fraction of the training data, can induce large language models to leak specific sensitive samples they have never encountered. To this end, the authors propose a novel data poisoning attack that injects carefully crafted poisoned examples to reshape the local loss landscape around the target output, creating sharp minima that force the model to memorize and subsequently leak the designated data. This approach is the first to exploit the geometric structure of the loss landscape to achieve targeted extraction of unseen data, revealing a viable bypass even under differential privacy protections. Experiments demonstrate 100% success in extracting target data from language models and 90% success from vision-language models, with effectiveness in both centralized and federated learning settings.

Abstract

Large Language Models are increasingly trained on proprietary or sensitive data, from private healthcare and financial records to user conversations containing secrets. Ensuring the privacy of such data against extraction attacks has become a central concern. In this paper, we ask whether an attacker who can poison a portion of the training data can facilitate the leakage of a separate target record they have no access to. We answer in the affirmative and show that such leakage can be induced by a poisoning mechanism that reshapes the model's local loss landscape around the target completion. Our key insight is that poisoning to create a sharp loss minimum at the target, surrounded by elevated loss on nearby alternatives, forces the model to memorize the target as the unique low-loss solution in its neighborhood. The attack requires no architectural changes, and generalizes across centralized and federated learning settings. We demonstrate that the attack amplifies privacy leakage across language (up to 100% successful extraction), and vision-language models (up 90% successful extraction). We show that the attack is thwarted when the model is trained to be differentially private. However, we introduce a new attack that directly probes the loss landscape bypassing even differential privacy defenses.

Problem

• data extraction
• training data privacy
• loss landscape
• model poisoning
• privacy attack

Innovation

• Loss Landscape Poisoning
• Training Data Extraction
• Privacy Attack
• Differential Privacy Bypass
• Memorization Induction