Prompt2Effect: Training-Free Image-to-Video Model Specialization via LoRA Generation

2026-06-11

AI Summary

Existing personalized image-to-video diffusion models require training a separate LoRA module for each visual effect, incurring substantial data and computational costs that hinder real-time interactivity. This work proposes Prompt2Effect—a training-free, weight-driven hypernetwork that generates LoRA weights corresponding to target visual effects in a single forward pass. The method innovatively conditions weight generation on the frozen base model architecture and introduces SVD-based normalized parameterization to resolve ambiguity in low-rank decomposition, significantly enhancing the accuracy and stability of high-dimensional weight synthesis. Experiments demonstrate that Prompt2Effect matches or surpasses conventional LoRA fine-tuning in video quality and effect alignment, reducing inference time from 56 GPU hours to 3.3 seconds; when used as a fine-tuning initializer, it accelerates optimization by approximately 10× while improving final performance.

Abstract

Personalizing Image-to-Video (I2V) diffusion models with specific visual effects is increasingly demanded for high-end video generation. Current practice requires training a separate Low-Rank Adaptation (LoRA) module for each effect, incurring substantial data curation and iterative optimization costs that hinder interactive control. We present Prompt2Effect, a weight-driven hypernetwork that amortizes per-effect training by directly synthesizing effect-specific LoRA weights in a single forward pass. Unlike prior hypernetworks that regress adapter weights purely from semantics, Prompt2Effect is explicitly conditioned on the frozen base model weights, grounding weight prediction in the structural geometry of each layer. Furthermore, instead of predicting raw LoRA matrices, we introduce an SVD-canonicalized parameterization that resolves factorization ambiguity and stabilizes large-scale weight synthesis. Together, these design principles enable accurate and scalable LoRA prediction for high-dimensional I2V diffusion models. Extensive experiments demonstrate that Prompt2Effect achieves on-par or superior video quality and effect alignment compared to conventional LoRA fine-tuning, while reducing the computational cost from 56 GPU training hours to 3.3 seconds of hypernetwork inference. When used as initialization for subsequent fine-tuning, our predicted weights further improve final performance and accelerate optimization by approximately 10x.

Problem

• Image-to-Video
• personalization
• LoRA
• diffusion models
• training-free

Innovation

• Prompt2Effect
• LoRA generation
• hypernetwork
• SVD-canonicalized parameterization
• image-to-video diffusion