SOYO: A Tuning-Free Approach for Video Style Morphing via Style-Adaptive Interpolation in Diffusion Models

2025-03-10

AI Summary

Existing video multi-style transfer methods often suffer from inter-frame structural incoherence and abrupt style transitions, hindering smooth temporal stylization. To address this, we propose a fine-tuning-free diffusion model framework that introduces the first style-adaptive sampling scheduler. By jointly leveraging attention injection and AdaIN normalization, our scheduler modulates intermediate features of a pre-trained text-to-image diffusion model to ensure structural consistency and temporally continuous style deformation. We further integrate an adaptive linear interpolation scheduling strategy, enabling zero-shot, high-quality multi-style transitions on open-domain videos. Experiments demonstrate that our method significantly outperforms state-of-the-art approaches across three key metrics: inter-frame structural consistency, style transition stability, and visual naturalness. This work establishes a novel paradigm for diffusion-based video stylization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Diffusion models have achieved remarkable progress in image and video stylization. However, most existing methods focus on single-style transfer, while video stylization involving multiple styles necessitates seamless transitions between them. We refer to this smooth style transition between video frames as video style morphing. Current approaches often generate stylized video frames with discontinuous structures and abrupt style changes when handling such transitions. To address these limitations, we introduce SOYO, a novel diffusion-based framework for video style morphing. Our method employs a pre-trained text-to-image diffusion model without fine-tuning, combining attention injection and AdaIN to preserve structural consistency and enable smooth style transitions across video frames. Moreover, we notice that applying linear equidistant interpolation directly induces imbalanced style morphing. To harmonize across video frames, we propose a novel adaptive sampling scheduler operating between two style images. Extensive experiments demonstrate that SOYO outperforms existing methods in open-domain video style morphing, better preserving the structural coherence of video frames while achieving stable and smooth style transitions.

Problem

• Seamless transitions in video style morphing
• Discontinuous structures in stylized video frames
• Imbalanced style morphing with linear interpolation

Innovation

• Uses pre-trained diffusion model without fine-tuning
• Combines attention injection and AdaIN techniques
• Introduces adaptive sampling scheduler for smooth transitions