FlowCompile: An Optimizing Compiler for Structured LLM Workflows

2026-05-13

AI Summary

This work addresses the challenge of efficiently balancing accuracy and latency in structured large language model (LLM) workflows, where the combinatorial design space—spanning model selection, inference budgets, and pipeline architecture—is prohibitively vast. To tackle this, the study introduces, for the first time, machine learning compilation principles into LLM workflow optimization. It performs global design space exploration prior to deployment by leveraging sub-agent decomposition, multi-configuration performance profiling, and a structure-aware surrogate model to accurately estimate and jointly optimize workflow-level accuracy and latency. The proposed approach generates a reusable set of Pareto-optimal configurations that span diverse accuracy–latency trade-offs, without requiring online adaptation or retraining. Experiments across multiple complex workflows and benchmarks demonstrate substantial improvements over heuristic and routing baselines, achieving up to 6.4× speedup while enabling flexible deployment and downstream scheduling.

Abstract

Structured LLM workflows, where specialized LLM sub-agents execute according to a predefined graph, have become a powerful abstraction for solving complex tasks. Optimizing such workflows, i.e., selecting configurations for each sub-agent to balance accuracy and latency, is challenging due to the combinatorial design space over model choices, reasoning budgets, and workflow structures. Existing cost-aware methods largely treat workflow optimization as a routing problem, selecting a configuration at inference time for each query according to the accuracy-latency objective used during training. We argue that structured LLM workflows can also be optimized from a compilation perspective: before deployment, the system can globally explore the workflow design space and construct a reusable set of workflow-level configurations spanning diverse accuracy-latency trade-offs. Drawing inspiration from machine learning compilers, we introduce FlowCompile, a structured LLM workflow compiler that performs compile-time design space exploration to identify a high-quality, reusable trade-off set. FlowCompile decomposes a workflow into sub-agents, profiles each sub-agent under diverse configurations, and composes these measurements through a structure-aware proxy to estimate workflow-level accuracy and latency. It then identifies diverse high-quality configurations in a single compile-time pass, without retraining or online adaptation. Experiments across diverse workflows and challenging benchmarks show that FlowCompile consistently outperforms heuristically optimized workflow configurations and routing-based baselines, delivering up to 6.4x speedup. The compiled configuration set further serves as a reusable optimization artifact, enabling flexible deployment under varying runtime preferences and supporting downstream selection or routing.

Problem

• structured LLM workflows
• workflow optimization
• accuracy-latency trade-off
• design space exploration
• compile-time optimization

Innovation

• structured LLM workflows
• compile-time optimization
• design space exploration
• accuracy-latency trade-off
• workflow compilation