FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving

2026-07-13

AI Summary

This work addresses the high latency and low throughput of diffusion model inference, which stem from multi-step repetitive updates across the entire latent space. Existing parallel approaches are hindered by substantial activation communication overhead. To overcome these limitations, the authors propose an adaptive region-based execution and scheduling framework that partitions semantically coherent regions guided by attention signals. A runtime controller selectively skips updates with minimal impact, while an affinity-aware online scheduler dynamically reallocates computational resources to jointly optimize load balancing and computation reuse. Evaluated across image, video, and audio generation tasks, the method reduces end-to-end latency by 30–97% and improves throughput by 1.2–2.2× compared to baseline approaches.

Abstract

Diffusion models have become the central backbone for modern image, video, and audio generation, but their efficient service remains a challenge. Unlike autoregressive decoding, diffusion inference repeatedly updates high-dimensional spatial or temporal latents over many denoising steps. This all-region execution pattern makes generation latency high and limits serving throughput. Existing multi-GPU parallelization methods can reduce per-step computation, but often introduce substantial activation exchange overhead, causing communication to offset or even outweigh the benefits of parallel execution. This paper presents FlashDiff, a diffusion serving system that improves inference efficiency through adaptive regional execution and scheduling. FlashDiff is based on the observation that diffusion refinement is not uniform across latent regions or denoising steps: different regions often stabilize at different rates, while neighboring steps exhibit strong temporal correlation. FlashDiff leverages these properties to selectively execute only regions that require further refinement and to reallocate the resulting compute slack across concurrent serving requests. FlashDiff consists of three mechanisms. First, it decomposes the latent representation into coherent execution regions using early-stage attention signals, preserving semantic structure while exposing fine-grained parallelism. Second, it uses a lightweight runtime controller to estimate region activity and bypass low-impact updates when further refinement is unlikely to affect output quality. Third, it applies an affinity-aware online scheduler that co-locates dependent regions, balances residual load across GPUs, and reuses reclaimed compute capacity to improve serving efficiency. Across real-world image, video, and audio workloads, FlashDiff reduces end-to-end serving latency by 30-97% and improves throughput by 1.2-2.2x.

Problem

• diffusion models
• inference efficiency
• serving latency
• multi-GPU parallelization
• activation exchange overhead

Innovation

• regional execution
• adaptive scheduling
• diffusion model serving
• compute reallocation
• latent refinement