GENSERVE: Efficient Co-Serving of Heterogeneous Diffusion Model Workloads

2026-04-05

AI Summary

This work addresses the challenge of co-serving heterogeneous diffusion models—such as text-to-image (T2I) and text-to-video (T2V)—on shared GPU clusters, where significant differences in computational demands, parallelization characteristics, and latency requirements often lead to severe violations of service-level objectives (SLOs). Leveraging the step-level predictability and preemptibility inherent in diffusion inference, the authors propose a novel co-optimization framework that integrates intelligent video preemption, elastic sequence parallelism, dynamic batching, and an SLO-aware scheduler to enable efficient resource allocation. This approach establishes a new paradigm for serving diverse generative workloads, achieving up to a 44% improvement in SLO compliance over the strongest baseline across various configurations while substantially enhancing both resource utilization and service quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Machine Learning: Hardware-aware ML

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Diffusion models have emerged as the prevailing approach for text-to-image (T2I) and text-to-video (T2V) generation, yet production platforms must increasingly serve both modalities on shared GPU clusters while meeting stringent latency SLOs. Co-serving such heterogeneous workloads is challenging: T2I and T2V requests exhibit vastly different compute demands, parallelism characteristics, and latency requirements, leading to significant SLO violations in existing serving systems. We present GENSERVE, a co-serving system that leverages the inherent predictability of the diffusion process to optimize serving efficiency. A central insight is that diffusion inference proceeds in discrete, predictable steps and is naturally preemptible at step boundaries, opening a new design space for heterogeneity-aware resource management. GENSERVE introduces step-level resource adaptation through three coordinated mechanisms: intelligent video preemption, elastic sequence parallelism with dynamic batching, and an SLO-aware scheduler that jointly optimizes resource allocation across all concurrent requests. Experimental results show that GENSERVE improves the SLO attainment rate by up to 44% over the strongest baseline across diverse configurations.

Problem

• diffusion models
• heterogeneous workloads
• co-serving
• latency SLOs
• GPU clusters

Innovation

• diffusion models
• co-serving
• step-level preemption
• elastic sequence parallelism
• SLO-aware scheduling