Training-Free Style and Content Transfer by Leveraging U-Net Skip Connections in Stable Diffusion 2.*

2025-01-24

AI Summary

Text-driven image editing suffers from poor content-style disentanglement and reliance on model fine-tuning. Method: We propose SkipInject, a zero-shot, training-free editing framework for Stable Diffusion 2. Leveraging the empirical finding that residual features from the third encoder block’s skip connection predominantly govern spatial structure reconstruction in the U-Net, SkipInject injects text-guided stylistic features at this layer and enforces explicit content-style separation via feature-level spatial alignment. Contribution/Results: This work is the first to systematically characterize the hierarchical distribution of spatial information across U-Net skip connections. SkipInject achieves high-fidelity structural preservation and precise semantic editing without any parameter updates. It outperforms state-of-the-art methods across diverse style transfer and content editing benchmarks, offering zero-training operation, computational efficiency, and strong user controllability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Structured Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite significant recent advances in image generation with diffusion models, their internal latent representations remain poorly understood. Existing works focus on the bottleneck layer (h-space) of Stable Diffusion's U-Net or leverage the cross-attention, self-attention, or decoding layers. Our model, SkipInject takes advantage of U-Net's skip connections. We conduct thorough analyses on the role of the skip connections and find that the residual connections passed by the third encoder block carry most of the spatial information of the reconstructed image, splitting the content from the style. We show that injecting the representations from this block can be used for text-based editing, precise modifications, and style transfer. We compare our methods state-of-the-art style transfer and image editing methods and demonstrate that our method obtains the best content alignment and optimal structural preservation tradeoff.

Problem

• Deep Learning
• U-Net Structure
• Image Transformation

Innovation

• SkipInject
• style-content separation
• U-Net skip connections