The More Popular, The Harder to Forget: Adaptive Popularity for LLM Unlearning

2026-08-14

AI Summary

This study addresses the limitation of existing large language model unlearning methods that overlook fact popularity, rendering high-frequency knowledge difficult to remove. We propose AdaPop, a novel approach that models popularity as a learnable parameter by integrating token confidence with external proxy evaluations. Through a bi-ascent controller, AdaPop dynamically adjusts penalty intensity to achieve an adaptive balance between forgetting and retention. Experiments across three model families and two benchmarks demonstrate that AdaPop reduces content leakage under paraphrased queries by approximately fivefold and under adversarial reconstruction by 1.6 times. Furthermore, internal representation analysis confirms superior forgetting separation, effectively resolving the challenge of unlearning high-frequency facts while preserving general model utility.

Abstract

Popular facts are memorised more deeply during pretraining and resist removal longer than rare ones, yet existing LLM unlearning methods apply uniform gradient pressure regardless of training-data frequency. We propose the AdaPop (Adaptive Popularity) method, which combines local token confidence with a per-fact popularity-dependent exponent derived from an external proxy (e.g., Wikidata sitelinks, LLM-as-Judge), and automates the forget-retain balance via a dual-ascent controller that adjusts the retain penalty each epoch. Across three model families and two benchmarks, AdaPop leaks ~5x less forgotten content than competing methods under paraphrased queries and ~1.6x less under adversarial reformulations. We support our analysis with internal metrics: under our method, forget-set hidden states move further from the pre-unlearning model's states than under other methods, while retain-set representations remain close.

Problem

• LLM Unlearning
• Adaptive Popularity
• Knowledge Removal
• Training Data Frequency

Innovation

• Adaptive Popularity
• LLM Unlearning
• Dual-ascent Controller
• Popularity-dependent Exponent
• Forget-retain Balance