Style Composition within Distinct LoRA modules for Traditional Art

2025-07-16

AI Summary

Existing diffusion models struggle with localized, multi-style fusion due to entangled latent representations and non-smooth interpolation, often resulting in global style dominance. To address this, we propose a zero-shot framework for regional multi-style composition: it employs LoRA modules to disentangle style representations and performs spatially masked feature stitching in the low-noise latent space. By integrating ControlNet-guided depth maps and flow-matching-based denoising, the method enables cross-model style combination and precise spatial control—without fine-tuning. Experiments demonstrate that our approach preserves high style fidelity across all constituent styles while supporting user-specified, fine-grained regional style blending. Both qualitative and quantitative evaluations show consistent superiority over state-of-the-art methods in terms of visual coherence, style accuracy, and spatial controllability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Diffusion-based text-to-image models have achieved remarkable results in synthesizing diverse images from text prompts and can capture specific artistic styles via style personalization. However, their entangled latent space and lack of smooth interpolation make it difficult to apply distinct painting techniques in a controlled, regional manner, often causing one style to dominate. To overcome this, we propose a zero-shot diffusion pipeline that naturally blends multiple styles by performing style composition on the denoised latents predicted during the flow-matching denoising process of separately trained, style-specialized models. We leverage the fact that lower-noise latents carry stronger stylistic information and fuse them across heterogeneous diffusion pipelines using spatial masks, enabling precise, region-specific style control. This mechanism preserves the fidelity of each individual style while allowing user-guided mixing. Furthermore, to ensure structural coherence across different models, we incorporate depth-map conditioning via ControlNet into the diffusion framework. Qualitative and quantitative experiments demonstrate that our method successfully achieves region-specific style mixing according to the given masks.

Problem

• Difficulty in controlled regional application of distinct painting techniques
• Lack of smooth interpolation between multiple artistic styles
• Entangled latent space causing one style to dominate

Innovation

• Zero-shot diffusion pipeline blends multiple styles
• Fuse latents across pipelines using spatial masks
• Depth-map conditioning ensures structural coherence