LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models

2026-07-06

AI Summary

This work addresses identity confusion and style interference in multi-concept personalized diffusion models by proposing a novel method that composes multiple independently trained LoRA modules without joint training. By modeling each concept as a distinct image layer, the approach employs a hierarchical LoRA architecture combined with a cascaded conditioning mechanism. With the backbone model frozen, individual LoRAs are activated sequentially, using the output of prior synthesis steps as conditional input, thereby effectively avoiding parameter-level interference. The method supports linear scalability and is agnostic to the underlying backbone, significantly enhancing identity preservation and visual consistency across multiple subjects. On Qwen-Image-Edit, it achieves an ArcFace identity detection rate of 0.861, markedly outperforming the baseline score of 0.745.

Abstract

Personalizing text-to-image diffusion models to render several specific subjects in a coherent image remains challenging: the model must preserve each subject's identity while keeping the scene spatially and visually coherent. Methods that fuse independently trained concept adapters in a shared weight space (via federated averaging, gradient fusion, or orthogonality constraints) suffer from identity confusion and style bleeding and require joint retraining. In this work, we show that composing concepts as separate image layers, instead of merging their adapters in a shared weight space, avoids parameter-level interference. We introduce LILAC, a framework that composes independently trained low-rank adapters at inference time: each subject is conditioned on the frozen composite of previously placed subjects, with exactly one adapter active at a time, therefore identities never interfere at the parameter level. LILAC composes the adapters without any joint training, scales linearly with the number of concepts, and is backbone-agnostic. Under the Orthogonal Adaptation protocol, LILAC applied on Qwen-Image-Edit reaches an ArcFace detection rate of 0.861, while Orthogonal Adaptation reports 0.745 in its original setting. Adaptation reports 0.745 in its original setting. Code is available at https://github.com/marianlupascu/LILAC.

Problem

• multi-concept customization
• identity preservation
• diffusion models
• style bleeding
• text-to-image generation

Innovation

• Layer-Wise Composition
• Independent LoRAs
• Cascaded Conditioning
• Parameter-Level Interference Avoidance
• Multi-Concept Customization