DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teacher

2026-01-29

AI Summary

This work addresses the challenges of unlearning harmful knowledge in large language models, which often involve high computational costs, catastrophic forgetting, or vulnerability to prompt-based attacks. The authors propose a distillation-based unlearning approach that guides a student model to mimic the behavior of a teacher model whose responses are modulated via contextual prompting. This method effectively rejects harmful content while preserving general capabilities, synergistically combining the strengths of both tuning-based and context-based unlearning paradigms. It achieves high efficiency, robustness, and data economy. Experimental results demonstrate that the proposed method significantly outperforms existing techniques across multiple benchmarks, exhibiting superior performance in unlearning efficacy, utility retention, and data efficiency—improving data efficiency by several orders of magnitude.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

LLM unlearning is a technique to remove the impacts of undesirable knowledge from the model without retraining from scratch, which is indispensable towards trustworthy AI. Existing unlearning methods face significant limitations: conventional tuning-based unlearning is computationally heavy and prone to catastrophic forgetting. In contrast, in-contextualized unlearning is lightweight for precise unlearning but vulnerable to prompt removal or reverse engineering attacks. In response, we propose Distilled Unlearning from an Efficient Teacher (DUET), a novel distillation-based unlearning method that combines the merits of these two lines of work. It learns a student model to imitate the behavior of a prompt-steered teacher that effectively refuses undesirable knowledge generation while preserving general domain knowledge. Extensive evaluations on existing benchmarks with our enriched evaluation protocols demonstrate that DUET achieves higher performance in both forgetting and utility preservation, while being orders of magnitude more data-efficient than state-of-the-art unlearning methods.

Problem

• LLM unlearning
• catastrophic forgetting
• in-context learning
• trustworthy AI
• knowledge removal

Innovation

• LLM unlearning
• knowledge distillation
• contextualized teacher
• data efficiency
• catastrophic forgetting