FATE: Future-State-Aware Scheduling for Heterogeneous LLM Workflows

2026-05-08

AI Summary

This work addresses the limitations of existing scheduling methods for heterogeneous large language model (LLM) workflows, which ignore future execution states and consequently suffer from increased end-to-end latency and fragmented useful states. To overcome these issues, the paper introduces a novel future-state-aware scheduling approach that, for the first time, treats future state preservation as a first-class optimization objective. It jointly models multiple dimensions of system state—including model residency, output locality, prefix reuse, and device reachability—through a CP-SAT-driven lookahead planner, horizon-aware candidate scoring, bounded multi-device sharded execution, and state-conditional cost estimation. Evaluated on real-world DAG benchmarks, the proposed method achieves normalized makespan and P95 latency of 0.675 and 0.677, respectively, representing improvements of approximately 8.9% and 8.8% over the strongest baseline and significantly outperforming existing strategies.

Abstract

Large language model (LLM) applications are increasingly executed as heterogeneous multi-stage workflows rather than isolated inference calls. In these workflow directed acyclic graphs (DAGs), scheduling decisions affect not only the currently ready stage, but also the execution state inherited by downstream stages, including model residency, parent-output locality, prefix reuse, and future device reachability. Existing serving and DAG-scheduling policies mainly optimize immediate queue state, placement cost, or reuse signals in isolation, which can fragment useful state and increase end-to-end latency. We present FATE, a future-state-aware scheduler for heterogeneous LLM workflows. FATE combines a CP-SAT-backed frontier planner, horizon-aware candidate scoring, bounded multi-device shard execution, and state-conditional cost estimation. Rather than solving a monolithic full-DAG problem, FATE repeatedly plans over the current ready frontier and scores assignments by both immediate cost and the downstream state they induce. Across real-DAG and controlled prefix-reuse benchmarks, FATE outperforms practical heuristics, classical DAG scheduling, and proxy adaptations of recent workflow-serving policies. On the real-DAG benchmark, it achieves normalized makespan and normalized P95 latency of 0.675 and 0.677, reducing them by 32.5% and 32.3% over RoundRobin and by 8.9% and 8.8% over the strongest non-FATE baseline. Mechanism analysis and ablations show that these gains arise from jointly preserving multiple dimensions of future execution state rather than prefix reuse alone. These results indicate that future-state preservation should be treated as a first-class scheduling objective for heterogeneous LLM workflow serving.

Problem

• LLM workflow scheduling
• future-state awareness
• heterogeneous execution
• DAG scheduling
• state preservation

Innovation

• future-state-aware scheduling
• heterogeneous LLM workflows
• DAG scheduling
• prefix reuse
• CP-SAT planning