Latency-Aware Orchestration for Multi-Agent LLM Workflows on Heterogeneous GPUs

2026-09-02

AI Summary

本文提出一种预测引导的运行时系统，通过预测GPU池状态来优化多代理LLM工作流的物理执行图，减少延迟并提高资源利用率。

Abstract

Concurrent multi-agent workflows expose future dependencies and serving-state requirements while running on heterogeneous GPU pools with time-varying load, model residency, and resource availability. The logical workflow defines the required computation, whereas its physical scheduling units, model-lifecycle actions, resource ordering, and placement must be selected according to the observed pool state. We present a prediction-guided runtime that uses workflow forecasts to construct and optimize a physical execution graph. Predictor estimates device-specific activation latency, peak memory, and model-loading cost, then propagates these predictions through workflow dependencies to forecast activation readiness and future model demand. Constructor builds semantics-preserving fusion and model-lifecycle alternatives, while Scheduler jointly optimizes their selection, placement, and execution order based on the live pool state. Across a workload spanning three workflow scenarios on a heterogeneous GPU pool, our system reduces end-to-end makespan and overall p95 completion latency under burst arrivals by up to 36.8% and 25.9%, respectively, over state-of-the-art workflow schedulers. It also saves up to 24.63 GPU-s per completed session.

Problem

• Latency-Aware Orchestration
• Multi-Agent LLM Workflows
• Heterogeneous GPUs
• Time-Varying Load
• Model Residency

Innovation

• prediction-guided runtime
• heterogeneous GPUs
• workflow forecast
• activation latency
• model lifecycle