🤖 AI Summary
This work addresses the challenge of simultaneously preserving structural stability and fine-grained style consistency in character generation using text-to-image diffusion models under no-reference conditions. To this end, we propose a lightweight cross-attention mechanism within the diffusion decoder that decouples and fuses textual semantics with learnable style embeddings. By fine-tuning only the decoder’s cross-attention layers and a compact style projection module, our approach achieves parameter-efficient, plug-and-play, and backbone-agnostic style control. Evaluated on the Pokémon character generation benchmark, the method significantly improves style fidelity, semantic alignment, and shape consistency while maintaining low computational overhead and inference simplicity.
📝 Abstract
This paper studies reference-free style-conditioned character generation in text-to-image diffusion models, where high-quality synthesis requires both stable character structure and consistent, fine-grained style expression across diverse prompts. Existing approaches primarily rely on text-only prompting, which is often under-specified for visual style and tends to produce noticeable style drift and geometric inconsistency, or introduce reference-based adapters that depend on external images at inference time, increasing architectural complexity and limiting deployment flexibility.We propose PokeFusion Attention, a lightweight decoder-level cross-attention mechanism that fuses textual semantics with learned style embeddings directly inside the diffusion decoder. By decoupling text and style conditioning at the attention level, our method enables effective reference-free stylized generation while keeping the pretrained diffusion backbone fully frozen.PokeFusion Attention trains only decoder cross-attention layers together with a compact style projection module, resulting in a parameter-efficient and plug-and-play control component that can be easily integrated into existing diffusion pipelines and transferred across different backbones.Experiments on a stylized character generation benchmark (Pokemon-style) demonstrate that our method consistently improves style fidelity, semantic alignment, and character shape consistency compared with representative adapter-based baselines, while maintaining low parameter overhead and inference-time simplicity.