Subject or Style: Adaptive and Training-Free Mixture of LoRAs

2025-08-04

AI Summary

Existing LoRA-based fine-tuning struggles to adaptively balance content fidelity and style generation, requiring laborious multi-hyperparameter tuning. To address this, we propose EST-LoRA—a training-free, adaptive LoRA fusion method. Its core innovation lies in the first-ever joint decision mechanism integrating an energy matrix, a style divergence score, and diffusion time-step information, which dynamically selects either the content or style LoRA branch per attention layer—inspired by Mixture-of-Experts (MoE) architectures. EST-LoRA requires no additional training or hyperparameter optimization, enabling seamless content-style disentanglement and synergistic generation. Extensive qualitative and quantitative evaluations demonstrate that EST-LoRA consistently outperforms state-of-the-art methods across diverse subject-style composition tasks, while offering faster inference and superior generalization.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Fine-tuning models via Low-Rank Adaptation (LoRA) demonstrates remarkable performance in subject-driven or style-driven generation tasks. Studies have explored combinations of different LoRAs to jointly generate learned styles and content. However, current methods struggle to balance the original subject and style, and often require additional training. Recently, K-LoRA proposed a training-free LoRA fusion method. But it involves multiple hyperparameters, making it difficult to adapt to all styles and subjects. In this paper, we propose EST-LoRA, a training-free adaptive LoRA fusion method. It comprehensively considers three critical factors: underline{E}nergy of matrix, underline{S}tyle discrepancy scores and underline{T}ime steps. Analogous to the Mixture of Experts (MoE) architecture, the model adaptively selects between subject LoRA and style LoRA within each attention layer. This integrated selection mechanism ensures balanced contributions from both components during the generation process. Experimental results show that EST-LoRA outperforms state-of-the-art methods in both qualitative and quantitative evaluations and achieves faster generation speed compared to other efficient fusion approaches. Our code is publicly available at: https://anonymous.4open.science/r/EST-LoRA-F318.

Problem

• Balancing subject and style in LoRA fusion without training
• Reducing hyperparameters in training-free LoRA fusion methods
• Adaptively selecting subject and style LoRA in attention layers

Innovation

• Training-free adaptive LoRA fusion method
• Considers energy, style scores, time steps
• Adaptively selects subject or style LoRA