FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models

2026-07-06

AI Summary

This work addresses the challenges of high memory consumption and low training efficiency in post-training large diffusion models, where existing parameter-efficient fine-tuning methods struggle to balance accuracy and efficiency. The authors propose FourTune, a framework that achieves the first end-to-end fully 4-bit (W4A4G4) post-training for diffusion models. FourTune enhances representational capacity through a three-branch hybrid LoRA architecture, isolates quantization-sensitive outliers via a frozen numerical stabilizer, and enables efficient backpropagation by integrating block-wise quantization with custom fused operators. Evaluated on FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25× and improves training throughput by 2.27× compared to BF16 LoRA, while preserving the generation quality of full-precision fine-tuning.

Abstract

Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25$\times$ and increases end-to-end training throughput by 2.27$\times$ compared to BF16 LoRA.

Problem

• diffusion models
• post-training
• memory efficiency
• training speed
• quantization

Innovation

• 4-bit quantization
• diffusion models
• parameter-efficient fine-tuning
• LoRA
• quantized backpropagation