FreeStyle: Free Control of Style-Content Dual-Reference Generation from Community LoRA Mining

2026-06-18

AI Summary

Existing dual-reference image generation methods struggle to simultaneously preserve content fidelity, achieve style alignment, and suppress semantic leakage from the style reference, further hindered by the absence of large-scale, stylistically diverse, and cleanly disentangled content-style triplets. This work proposes FreeStyle, a novel framework that first leverages community-trained LoRAs to construct a scalable dataset of Style-Reference/Content-Reference triplets. It then introduces a two-stage curriculum learning strategy augmented with attention-level enrichment constraints and a frequency-aware RoPE modulation mechanism to effectively mitigate semantic leakage. Finally, a new evaluation benchmark is established by integrating content alignment scores with calibrated VLM rejection scores. Experiments demonstrate that FreeStyle significantly outperforms existing approaches in both style transfer and dual-reference generation, achieving an optimal balance among content preservation, style alignment, and leakage suppression.

Abstract

Style-content dual-reference generation aims to synthesize an image that preserves the structure and semantics of a content reference while adopting the style of a separate style reference.Despite recent progress, this setting remains challenging because models must balance content fidelity, style alignment, and instruction following avoiding semantic leakage from the style reference.A key bottleneck is the lack of large-scale triplet data with clean content-style separation and broad long-tail style coverage.In this work, we propose FreeStyle, a scalable dual-reference generation framework based on community LoRA mining.We treat community LoRAs as compositional anchors for style and content, and design a rigorous generation and filtering pipeline to construct large-scale Style-Reference and Content-Reference triplets across multiple base models.To address content leakage, we adopt a two-stage curriculum with stage-specific disentanglement mechanisms: an attention-level enrichment constraint that suppresses style-reference leakage in the style-transfer stage, and a frequency-aware RoPE modulation strategy that targets positional-correspondence-based leakage in the harder dual-reference stage.We also introduce a benchmark covering both style-reference and dual-reference generation, with evaluations on style similarity, content preservation, aesthetics, instruction following, and leakage rejection. The benchmark incorporates a style-invariant Content Alignment Score (CAS) and introduces a calibrated VLM-based Rejection Score for evaluating generation reliability and leakage suppression.Extensive experiments show that our model achieves a strong balance among style alignment, content preservation, and leakage suppression.

Problem

• style-content dual-reference generation
• semantic leakage
• content fidelity
• style alignment
• instruction following

Innovation

• dual-reference generation
• LoRA mining
• semantic leakage suppression
• style-content disentanglement
• curriculum learning