AnyStyle: A Single LoRA is Sufficient for Image-Guided Style Transfer

2026-07-06

AI Summary

Existing image-guided style transfer methods suffer from low structural fidelity and difficulty in balancing content and style during inference, primarily due to the tight coupling between content and style representations and conflicts arising from multiple adapter modules. To address these limitations, this work proposes AnyStyle, a novel framework that abandons multi-adapter designs and instead introduces a single LoRA module to unify style modeling. Leveraging the internal self-attention mechanisms of pre-trained diffusion models, AnyStyle extracts content structure guidance signals without requiring additional training. This approach significantly enhances controllability, stability, and computational efficiency during style transfer. While maintaining competitive quantitative performance, AnyStyle achieves more accurate structure preservation and higher-quality artistic stylization compared to existing methods.

Abstract

Image-guided style transfer aims to apply the artistic characteristics of a style image to a content image while preserving its semantic structure and layout. Despite advances in diffusion-based methods, existing approaches often face challenges in disentangling content and style, particularly when independently optimized adapters are naively combined, causing conflicts between adapters and limiting controllability over the content-style balance in inference. We further demonstrate that training-free structural guidance directly derived from the content image through the internal attention of pre-trained model outperforms a dedicated content LoRA adapter in terms of structural fidelity and computational efficiency. Building on these observations, we propose AnyStyle, a streamlined framework for image-guided style transfer. The framework adopts a unified single-adapter paradigm for coherent style capture from the style image and incorporates training-free structural guidance from the content image, thus avoiding complex entanglement between multiple adapters and improving controllability and stability. Extensive experiments show that our method delivers competitive quantitative performance and significantly improved perceptual quality. Code is available at https://github.com/Yvan1001/AnyStyle.

Problem

• style transfer
• content-style disentanglement
• diffusion models
• adapter conflict
• controllability

Innovation

• Single LoRA
• Training-free structural guidance
• Image-guided style transfer
• Content-style disentanglement
• Diffusion models