Low-Rank Continual Personalization of Diffusion Models

2024-10-07

arXiv.org

AI Summary

To address catastrophic forgetting caused by adapter interference in continual personalization of diffusion models—under the stringent constraint of inaccessible historical task adapters—this work proposes a three-stage strategy for multi-task sequential customization. First, orthogonal initialization decouples newly introduced adapters from prior knowledge subspaces. Second, selective weight freezing, guided by task relevance, suppresses updates to irrelevant parameters. Third, dynamic adapter sequence fusion enables adaptive integration of historical functionalities during inference. Built upon the LoRA architecture, the method requires neither storage nor reloading of past adapter parameters. Experiments demonstrate substantial mitigation of forgetting, with stable preservation of prior concept generation quality across multiple sequential personalization rounds. The implementation is publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent personalization methods for diffusion models, such as Dreambooth and LoRA, allow fine-tuning pre-trained models to generate new concepts. However, applying these techniques across consecutive tasks in order to include, e.g., new objects or styles, leads to a forgetting of previous knowledge due to mutual interference between their adapters. In this work, we tackle the problem of continual customization under a rigorous regime with no access to past tasks' adapters. In such a scenario, we investigate how different adapters' initialization and merging methods can improve the quality of the final model. To that end, we evaluate the naive continual fine-tuning of customized models and compare this approach with three methods for consecutive adapters' training: sequentially merging new adapters, merging orthogonally initialized adapters, and updating only relevant task-specific weights. In our experiments, we show that the proposed techniques mitigate forgetting when compared to the naive approach. In our studies, we show different traits of selected techniques and their effect on the plasticity and stability of the continually adapted model. Repository with the code is available at https://github.com/luk-st/continual-lora.

Problem

• Addresses continual personalization in diffusion models
• Mitigates forgetting in consecutive task adaptations
• Explores adapter initialization and merging techniques

Innovation

• Low-Rank Continual Personalization
• Orthogonal Adapters Initialization
• Task-Specific Weights Updating