Selective, Controlled and Domain-Agnostic Unlearning in Pretrained CLIP: A Training- and Data-Free Approach

2025-12-16

AI Summary

Addressing the challenge of selectively forgetting category-specific knowledge—such as global class removal, domain-specific erasure, or intra-domain complete forgetting—in pretrained multimodal models like CLIP under zero-shot settings—without access to retraining or original training data—this paper proposes the first training- and data-free zero-space forgetting framework. Leveraging CLIP’s joint embedding space, our method constructs text-guided synthetic visual prototypes and a multimodal null space, enabling controllable knowledge masking via directional projection. It supports three fine-grained forgetting paradigms, eliminating reliance on retraining. Evaluated across multiple vision domains, the framework achieves an average forgetting rate of 92.3%, while inducing only a 0.76% drop in accuracy on retained tasks—demonstrating near-lossless performance.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Pretrained models like CLIP have demonstrated impressive zero-shot classification capabilities across diverse visual domains, spanning natural images, artistic renderings, and abstract representations. However, real-world applications often demand the removal (or "unlearning") of specific object classes without requiring additional data or retraining, or affecting the model's performance on unrelated tasks. In this paper, we propose a novel training- and data-free unlearning framework that enables three distinct forgetting paradigms: (1) global unlearning of selected objects across all domains, (2) domain-specific knowledge removal (e.g., eliminating sketch representations while preserving photo recognition), and (3) complete unlearning in selective domains. By leveraging a multimodal nullspace through synergistic integration of text prompts and synthesized visual prototypes derived from CLIP's joint embedding space, our method efficiently removes undesired class information while preserving the remaining knowledge. This approach overcomes the limitations of existing retraining-based methods and offers a flexible and computationally efficient solution for controlled model forgetting.

Problem

• Removes specific object classes without data or retraining
• Enables domain-specific or global knowledge forgetting selectively
• Preserves model performance on unrelated tasks efficiently

Innovation

• Training- and data-free unlearning framework
• Multimodal nullspace with text prompts and visual prototypes
• Selective, controlled, domain-agnostic forgetting paradigms