Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning

2024-10-09

arXiv.org

AI Summary

This work addresses the instability of gradient-ascent-based methods in large language model (LLM) unlearning, which often suffer from divergence, excessive forgetting, and catastrophic performance degradation. To this end, we propose SimNPO—a reference-free framework that integrates simplified preference optimization, gradient magnitude control, and Markov chain state modeling to enable stable, reference-free unlearning while avoiding reference-model bias. Evaluated on the TOFU and MUSE benchmarks, SimNPO achieves superior unlearning precision over state-of-the-art approaches and demonstrates strong robustness against relearning attacks. The implementation is publicly available.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This work studies the problem of large language model (LLM) unlearning, aiming to remove unwanted data influences (e.g., copyrighted or harmful content) while preserving model utility. Despite the increasing demand for unlearning, a technically-grounded optimization framework is lacking. Gradient ascent (GA)-type methods, though widely used, are suboptimal as they reverse the learning process without controlling optimization divergence (i.e., deviation from the pre-trained state), leading to risks of over-forgetting and potential model collapse. Negative preference optimization (NPO) has been proposed to address this issue and is considered one of the state-of-the-art LLM unlearning approaches. In this work, we revisit NPO and identify another critical issue: reference model bias. This bias arises from using the reference model (i.e., the model prior to unlearning) to evaluate the unlearning success, which can compromise NPO's effectiveness. Specifically, it leads to (a) uneven allocation of optimization power across forget data with varying difficulty levels and (b) ineffective gradient weight smoothing during the early stages of unlearning optimization. To overcome these challenges, we propose a simple yet effective unlearning optimization framework, called SimNPO, showing that `simplicity' in removing the reliance on a reference model (through the lens of simple preference optimization) benefits unlearning. We provide deeper insights into SimNPO's advantages through an analysis based on mixtures of Markov chains. Extensive experiments further validate SimNPO's efficacy on benchmarks like TOFU and MUSE, as well as its robustness against relearning attacks. Codes are available at https://github.com/OPTML-Group/Unlearn-Simple.

Problem

• Optimize LLM unlearning to remove unwanted data influences.
• Address reference model bias in negative preference optimization.
• Propose SimNPO for simpler, effective unlearning without reference model.

Innovation

• SimNPO removes reference model bias
• SimNPO ensures effective gradient weight smoothing
• SimNPO simplifies unlearning optimization framework