RAZOR: Ratio-Aware Layer Editing for Targeted Unlearning in Vision Transformers and Diffusion Models

2026-03-16

AI Summary

This work addresses the challenge of efficiently and precisely unlearning sensitive or undesirable content in vision Transformers and diffusion models without requiring full retraining. The authors propose a lightweight, model-agnostic editing framework that enables targeted forgetting by jointly and proportionally updating critical layers and attention heads within the Transformer backbone. Their approach identifies target components through contribution-aware evaluation and applies a regularized update rule that minimizes modification magnitude while preserving essential knowledge. Evaluated on large vision models such as CLIP and Stable Diffusion across identity, style, and object erasure tasks, the method consistently outperforms existing techniques in terms of forgetting accuracy, retention of useful knowledge, computational efficiency, and robustness under quantitative evaluation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Transformer based diffusion and vision-language models have achieved remarkable success; yet, efficiently removing undesirable or sensitive information without retraining remains a central challenge for model safety and compliance. We introduce Ratio-Aware Zero/One-step Optimized Retentive unlearning (RAZOR), a lightweight, model-agnostic unlearning framework that generalizes forgetting updates to coordinated multi-layer and multi-head edits within transformer backbones. RAZOR identifies the most important layers and attention heads by measuring how much they contribute to forgetting the target data while preserving useful knowledge. Then, it updates these parts of the model using a carefully regularized rule to avoid harming overall performance. The set of edited components grows gradually, ensuring precise unlearning without over-editing or damaging unrelated capabilities. We evaluate RAZOR on CLIP, Stable Diffusion, and vision-language models (VLMs) using widely adopted unlearning benchmarks covering identity, style, and object erasure tasks. Our results show that RAZOR achieves highly accurate and stable forgetting, even under quantization. This approach offers stronger retention and better efficiency than prior methods. Notably, it also operates significant faster than conventional techniques. These results demonstrate that RAZOR is a practical and scalable solution for safe, adaptive unlearning in transformer-based vision models.

Problem

• machine unlearning
• vision transformers
• diffusion models
• model editing
• data removal

Innovation

• machine unlearning
• vision transformers
• diffusion models
• attention editing
• model quantization