Compass: Optimizing Compound AI Workflows for Dynamic Adaptation

2026-03-21

AI Summary

This work addresses the challenge of simultaneously meeting service-level objectives (SLOs) for accuracy, latency, and cost in composite AI systems under dynamic workloads within fixed infrastructure. The authors propose the first framework that enables runtime dynamic configuration switching by integrating offline optimization with online adaptation to intelligently balance accuracy and latency. The framework incorporates the COMPASS-V algorithm—combining finite-difference-guided search, hill climbing, and horizontal scaling—a queueing-theory-based policy planner, and the Elastico runtime controller. Experimental results demonstrate that the configuration discovery phase reduces evaluation overhead by 57.5% on average (up to 95.3%), achieves runtime SLO compliance rates of 90–98%, improves SLO adherence by 71.6% over static high-accuracy baselines, and enhances accuracy by 3–5% compared to static low-latency baselines.

Technology Category

• Search and Optimization: Algorithm Configuration
• Machine Learning: Hardware-aware ML
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Compound AI is a distributed intelligence approach that represents a unified system orchestrating specialized AI/ML models with engineered software components into AI workflows. Compound AI production deployments must satisfy accuracy, latency, and cost objectives under varying loads. However, many deployments operate on fixed infrastructure where horizontal scaling is not viable. Existing approaches optimize solely for accuracy and do not consider changes in workload conditions. We observe that compound AI systems can switch between configurations to fit infrastructure capacity, trading accuracy for latency based on current load. This requires discovering multiple Pareto-optimal configurations from a combinatorial search space and determining when to switch between them at runtime. We present Compass, a novel framework that enables dynamic configuration switching through offline optimization and online adaptation. Compass consists of three components: COMPASS-V algorithm for configuration discovery, Planner for switching policy derivation, and Elastico Controller for runtime adaptation. COMPASS-V discovers accuracy-feasible configurations using finite-difference guided search and a combination of hill-climbing and lateral expansion. Planner profiles these configurations on target hardware and derives switching policies using a queuing theory based model. Elastico monitors queue depth and switches configurations based on derived thresholds. Across two compound AI workflows, COMPASS-V achieves 100% recall while reducing configuration evaluations by 57.5% on average compared to exhaustive search, with efficiency gains reaching 95.3% at tight accuracy thresholds. Runtime adaptation achieves 90-98% SLO compliance under dynamic load patterns, improving SLO compliance by 71.6% over static high-accuracy baselines, while simultaneously improving accuracy by 3-5% over static fast baselines.

Problem

• Compound AI
• Dynamic Adaptation
• Accuracy-Latency Tradeoff
• Workload Variability
• SLO Compliance

Innovation

• Compound AI
• Dynamic Adaptation
• Pareto-optimal Configuration
• Runtime Optimization
• Queue-aware Scheduling