ICCU: In-Context Continual Unlearning via Pattern-Induced Refusal Rules

2026-05-26

AI Summary

This work addresses the high computational cost, significant utility degradation, and cross-request interference associated with conventional fine-tuning approaches to continual unlearning in language models. To overcome these limitations, the authors propose a parameter-free, context-based continual unlearning framework that induces human-readable, order-agnostic refusal rules from data designated for forgetting. These rules are dynamically applied during inference either as filters or system prompts, and the original forgettable data can be discarded after rule generation, thereby preserving privacy. Leveraging pattern-induced rule synthesis, contextual inference control, and a rule composition mechanism, the method effectively suppresses targeted knowledge while maintaining model utility, demonstrates strong scalability across multiple sequential unlearning requests, and exhibits robustness against query rephrasing and cross-lingual inputs.

Abstract

Machine unlearning aims to remove the influence of specific data from trained language models. In real-world deployments, unlearning requests often arrive sequentially, which challenges existing fine-tuning-based methods: fine-tuning each request is costly, accumulates utility loss, and may cause cross-request interference. To address these issues, we propose ICCU (In-Context Continual Unlearning), an in-context continual unlearning framework that induces readable refusal rules from unlearning datasets and applies them at inference time either as a filter or via the system prompt, without modifying model parameters. Because rules are accumulated as an order-independent union, ICCU is compositional and free of cross-request interference, and the original forget-set data can be discarded after rule induction. Extensive experiments show that ICCU effectively suppresses target knowledge while preserving utility, scales across sequential requests, and remains robust to paraphrased and cross-lingual queries.

Problem

• machine unlearning
• continual unlearning
• sequential unlearning requests
• cross-request interference
• utility loss

Innovation

• machine unlearning
• in-context learning
• refusal rules
• continual unlearning
• parameter-free