DuoLoRA : Cycle-consistent and Rank-disentangled Content-Style Personalization

2025-04-15

AI Summary

Existing few-shot personalization methods treat content and style as independent variables, overlooking their inherent semantic coupling and thus limiting joint modeling performance. To address this, we propose a decoupled rank-dimensional co-optimization framework. Our approach introduces three key innovations: (i) ZipRank—a rank-dimensional masking learning mechanism; (ii) SDXL layer-prior-driven adaptive fusion initialization; and (iii) Constyle—a cycle-consistency loss enforcing semantic alignment between content and style in a low-rank subspace. Built upon the LoRA architecture, our method leverages layer-aware prior guidance and cyclic consistency constraints to transcend the conventional independence assumption. Extensive evaluations across multiple benchmarks demonstrate significant improvements over state-of-the-art methods—including ZipLoRA—with over 40% reduction in trainable parameters, while simultaneously enhancing content fidelity and style transfer quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Personalized content ranking and presentation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We tackle the challenge of jointly personalizing content and style from a few examples. A promising approach is to train separate Low-Rank Adapters (LoRA) and merge them effectively, preserving both content and style. Existing methods, such as ZipLoRA, treat content and style as independent entities, merging them by learning masks in LoRA's output dimensions. However, content and style are intertwined, not independent. To address this, we propose DuoLoRA, a content-style personalization framework featuring three key components: (i) rank-dimension mask learning, (ii) effective merging via layer priors, and (iii) Constyle loss, which leverages cycle-consistency in the merging process. First, we introduce ZipRank, which performs content-style merging within the rank dimension, offering adaptive rank flexibility and significantly reducing the number of learnable parameters. Additionally, we incorporate SDXL layer priors to apply implicit rank constraints informed by each layer's content-style bias and adaptive merger initialization, enhancing the integration of content and style. To further refine the merging process, we introduce Constyle loss, which leverages the cycle-consistency between content and style. Our experimental results demonstrate that DuoLoRA outperforms state-of-the-art content-style merging methods across multiple benchmarks.

Problem

• Jointly personalizing content and style from few examples
• Merging content and style adaptively with rank flexibility
• Enhancing integration via cycle-consistency and layer priors

Innovation

• Rank-dimension mask learning for merging
• SDXL layer priors enhance integration
• Constyle loss ensures cycle-consistency