Cost-Aware Routing for Efficient Text-To-Image Generation

2025-06-17

AI Summary

To address the trade-off between high-fidelity generation and excessive computational cost in diffusion-based text-to-image synthesis, this paper proposes a prompt-aware dynamic multi-model routing framework. Methodologically, it first quantifies prompt complexity and then employs a learned routing policy to dynamically dispatch prompts of varying complexity to the most suitable generative model—such as diffusion models with differing step counts or lightweight distilled variants—enabling on-demand allocation of computational resources. Its key innovation lies in being the first framework to support adaptive, coordinated scheduling of nine heterogeneous pre-trained models within a unified architecture, thereby transcending rigid budget-constrained paradigms. Experiments on COCO and DiffusionDB demonstrate that our method achieves superior average generation quality over all baselines while significantly reducing overall FLOPs. It preserves high fidelity for complex prompts and accelerates inference for simple prompts by over 3×.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Diffusion models are well known for their ability to generate a high-fidelity image for an input prompt through an iterative denoising process. Unfortunately, the high fidelity also comes at a high computational cost due the inherently sequential generative process. In this work, we seek to optimally balance quality and computational cost, and propose a framework to allow the amount of computation to vary for each prompt, depending on its complexity. Each prompt is automatically routed to the most appropriate text-to-image generation function, which may correspond to a distinct number of denoising steps of a diffusion model, or a disparate, independent text-to-image model. Unlike uniform cost reduction techniques (e.g., distillation, model quantization), our approach achieves the optimal trade-off by learning to reserve expensive choices (e.g., 100+ denoising steps) only for a few complex prompts, and employ more economical choices (e.g., small distilled model) for less sophisticated prompts. We empirically demonstrate on COCO and DiffusionDB that by learning to route to nine already-trained text-to-image models, our approach is able to deliver an average quality that is higher than that achievable by any of these models alone.

Problem

• Balancing quality and computational cost in text-to-image generation
• Automatically routing prompts to optimal text-to-image models
• Reducing computational expense for simpler prompts while maintaining quality

Innovation

• Dynamic routing based on prompt complexity
• Utilizes multiple pre-trained text-to-image models
• Optimizes computation-cost and quality trade-off