Exploring Data-Free LoRA Transferability for Video Diffusion Models

2026-05-03

AI Summary

Existing LoRA methods struggle to transfer directly to video diffusion models based on step distillation or causal distillation, often leading to style degradation and structural collapse. This work identifies the root cause as spectral interference within shared functional clusters in weight space and introduces Cluster-Aware Spectral Arbitration (CASA), a training-data-free framework. CASA leverages singular subspace analysis of functional clusters and a spectral density-aware dynamic arbitration mechanism to align LoRA adaptations effectively while preserving the target manifold geometry. Experiments demonstrate that CASA substantially mitigates transfer artifacts, restores LoRA functionality, and exhibits strong generality and effectiveness across diverse video diffusion architectures.

Abstract

Video diffusion models leveraging step distillation or causal distillation have achieved remarkable performance. However, adapting existing LoRAs to these variants remains a critical challenge due to weight space mismatches. We observe that direct application leads to style degradation and structural collapse, yet the underlying mechanisms remain poorly understood. To fill this gap, we delve into the weight space and identify that the incompatibility stems from spectral interference within shared functional clusters defined over singular subspaces. Specifically, our analysis reveals that while both paradigms respect spectral rigidity, they establish conflicting routing pathways that clash through constructive overload or destructive cancellation. To address this issue, we propose Cluster-Aware Spectral Arbitration (CASA), a data-free framework that dynamically arbitrates between safeguarding the target's manifold and restoring LoRA alignment based on spectral density. Extensive experiments demonstrate that CASA effectively mitigates artifacts and revives LoRA functionality. Our code is available at https://github.com/Noahwangyuchen/CASA

Problem

• LoRA transferability
• video diffusion models
• weight space mismatch
• spectral interference
• data-free adaptation

Innovation

• LoRA transferability
• video diffusion models
• spectral interference
• data-free adaptation
• singular subspaces