Natively Unlearnable Large Language Models

2026-06-11

AI Summary

This work addresses the challenge of efficiently and precisely unlearning specific training data sources in large language models while preserving the rest of the learned knowledge. The authors propose the NULLs architecture, which employs a shared backbone coupled with sparse, source-specific “sinks” that are co-trained to concentrate source-unique information within their respective sinks while retaining general knowledge in the backbone. This approach uniquely unifies source-level unlearnability with joint representation learning, enabling exact forgetting—without retraining or access to original data—simply by deactivating the corresponding sink. The method demonstrates robustness against adversarial extraction and relearning attacks. Experiments on a corpus of approximately six million Wikipedia articles show that forgetting a single article achieves performance nearly equivalent to full retraining from scratch, while maintaining downstream task performance on par with standard Transformers.

Abstract

Unlearning aims to remove the influence of specific training data sources, but this has proved challenging because the contributions of different sources are entangled within the model. Isolating source contributions to disjoint parameters makes removal easier, though it obstructs joint learning across sources. We propose NULLs (Natively Unlearnable LLMs), a model class that satisfies the two opposing goals of isolating source-specific contributions and learning jointly across sources, by training a set of shared backbone neurons alongside a pool of sparsely activated sinks. During training, information specific to a source naturally concentrates in its sinks while information shared across sources accumulates in the backbone. A source is then unlearned at deployment by disabling its corresponding sinks, with no gradient updates and no access to the retained data. We show that NULLs scales to Wikipedia's ~6M articles, isolating each as an independent source. Unlearning a single article removes knowledge specific to it while preserving facts shared with semantically related articles, closely matching retraining from scratch. We note that unlearning with NULLs is also robust: in a case study of unlearning the Harry Potter books, NULLs resists both adversarial extraction and relearning that reverses post-hoc unlearning. Finally, NULLs preserves general language capabilities, matching a standard transformer on downstream benchmarks. Together, these results suggest that source-level unlearning need not be an afterthought. It can be built natively into LLM training while retaining the benefits of shared representation learning.

Problem

• unlearning
• large language models
• data removal
• source isolation
• model forgetting

Innovation

• unlearning
• large language models
• modular architecture
• source isolation
• sparse activation