Continual Personalization for Diffusion Models

2025-10-02

AI Summary

Diffusion models suffer from catastrophic forgetting during incremental learning for personalization, degrading zero-shot text-to-image generation performance. Method: We propose an efficient continual personalization framework that introduces a concept neuron selection mechanism to precisely identify and fine-tune a sparse subset of neurons strongly associated with new concepts. It integrates incremental fine-tuning with joint knowledge preservation, maintaining both historical concept fidelity and new concept adaptation using minimal parameter updates. The approach requires no model ensembling or architectural modifications and supports both single- and multi-concept incremental learning. Contribution/Results: Our method achieves state-of-the-art performance on standard benchmarks, significantly outperforming existing approaches while reducing memory overhead and inference latency. Crucially, it is the first to effectively reconcile high-fidelity zero-shot generation with continual personalization—enabling robust, scalable, and resource-efficient customization without compromising generative capability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Life-Long and Continual Learning
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Updating diffusion models in an incremental setting would be practical in real-world applications yet computationally challenging. We present a novel learning strategy of Concept Neuron Selection (CNS), a simple yet effective approach to perform personalization in a continual learning scheme. CNS uniquely identifies neurons in diffusion models that are closely related to the target concepts. In order to mitigate catastrophic forgetting problems while preserving zero-shot text-to-image generation ability, CNS finetunes concept neurons in an incremental manner and jointly preserves knowledge learned of previous concepts. Evaluation of real-world datasets demonstrates that CNS achieves state-of-the-art performance with minimal parameter adjustments, outperforming previous methods in both single and multi-concept personalization works. CNS also achieves fusion-free operation, reducing memory storage and processing time for continual personalization.

Problem

• Personalizes diffusion models incrementally to prevent catastrophic forgetting
• Identifies concept-related neurons for efficient multi-concept personalization
• Enables fusion-free continual learning with minimal parameter adjustments

Innovation

• Concept Neuron Selection identifies target-related neurons
• Incremental finetuning prevents catastrophic forgetting issues
• Fusion-free operation reduces memory and processing requirements