Principal-timestep Restricted Init via Sparse Matrix-decomposition in Flow-matching

2026-09-14

AI Summary

该研究解决了流匹配扩散模型微调成本高的问题，通过提出Prism-LoRA方法，利用主时间步选择和主通道过滤提高梯度对齐，加速收敛并提升性能。

Abstract

Flow-matching diffusion models have recently emerged as a strong paradigm for high-fidelity visual generation. However, their prohibitively high fine-tuning cost limits scalability to downstream tasks. While Low-Rank Adaptation (LoRA) combined with spectral initialization has demonstrated accelerated convergence and improved performance in autoregressive language models by better aligning gradient directions, we find that it fails to deliver similar gains in diffusion fine-tuning, often yielding marginal or even negative improvements over vanilla LoRA.We attribute this discrepancy to a fundamental mismatch between LoRA's low-rank parameterization and the intrinsically high-rank gradients induced by the flow-matching objective. In particular, stochastic timestep sampling introduces directionally heterogeneous gradient signals across training steps, leading to misaligned updates under low-rank constraints.To address this issue, we propose Prism-LoRA,a Principal-timestep Restricted Init via Sparse Matrix-decomposition framework that improves gradient alignment during fine-tuning. Our method consists of two key components: (i) principal timestep selection, which restricts initialization gradients to a subset of dominant timesteps to suppress effective gradient rank, and (ii) principal channel filtering, which removes task-irrelevant channels, enabling the one-step spectral initialization gradient to better align with the long-horizon optimization trajectory. Extensive experiments demonstrate that our method consistently improves both convergence speed and final performance across multiple diffusion fine-tuning benchmarks, including subject-driven generation, controllable generation, and deblurring, achieving not only performance improvement but also earlier stages of convergence over baseline LoRA and other spectral-init methods.

Problem

• Flow-matching
• Diffusion models
• Fine-tuning cost
• Low-Rank Adaptation (LoRA)
• Gradient alignment

Innovation

• Principal-timestep Selection
• Sparse Matrix-decomposition
• Gradient Alignment
• Flow-matching Diffusion Models
• Low-Rank Adaptation