A Workflow-Aware Serving Layer for Agentic Applications

2026-07-03

AI Summary

This work addresses the lack of workflow-awareness in existing model serving engines and the inability of agent frameworks to jointly optimize model selection, verifier configuration, and backend execution under varying loads. To bridge this gap, we propose Dyserve—the first workflow-aware serving layer tailored for agent applications—that uniquely integrates workflow topology, model choices, and heterogeneous backend resources into a unified optimization framework. By formulating global scheduling as an integer linear program and introducing a quality allocation mechanism based on error propagation depth, Dyserve enables coordinated decision-making across the serving stack. Moreover, it supports runtime, solver-free adaptive policy switching and residual replanning, achieving significant improvements in resource utilization and response efficiency while maintaining stringent service quality guarantees.

Abstract

Agentic AI applications form an emerging serving workload in which a request creates a workflow: a directed acyclic graph of LLM and tool calls that exposes per-node model choices and optional quality operators such as verifiers. This workload falls between two existing layers. Model-serving engines execute individual calls efficiently but cannot see workflow structure, while agent frameworks fix the workflow but cannot see backend load, so neither jointly chooses each node's model, verifier, and backend under serving-time conditions. We present Dyserve, a workflow-aware serving layer that fills this gap. Dyserve compiles each workflow's per-node model and verifier choices in one integer linear program (ILP) over a heterogeneous backend pool, priced by skill-conditioned offline profiles that transfer across workflows. This couples with hardware entering only through per-model throughput sweeps, and is weighted to concentrate strong models and verification on the nodes whose errors propagate the furthest. Because no single latency-quality preference fits every workload mix, Dyserve pre-solves the program at several pressure levels at admission and shifts a workflow's uncommitted suffix among these strategies under load, keeping the solver off the load-shift path; a failed tool call triggers a one-time residual re-solve that preserves committed work.

Problem

• Agentic AI
• workflow-aware serving
• model selection
• quality verification
• serving optimization

Innovation

• workflow-aware serving
• integer linear programming
• agentic AI
• heterogeneous backend
• dynamic load adaptation