HEXGEN-TEXT2SQL: Optimizing LLM Inference Request Scheduling for Agentic Text-to-SQL Workflow

2025-05-08

AI Summary

Existing LLM serving frameworks struggle to jointly handle task dependencies, dynamic latency fluctuations, and GPU hardware heterogeneity in production-grade, multi-stage, low-latency Text-to-SQL systems—leading to frequent SLO violations and inefficient scheduling. Method: We propose a hierarchical scheduling architecture comprising global load-balanced request dispatching and local adaptive urgency-aware scheduling. We introduce the first simulation-driven lightweight online hyperparameter optimization method, integrated with workflow feature modeling, multi-stage dependency graph management, and heterogeneous-GPU-aware dynamic priority assignment. Results: Evaluated on a realistic Text-to-SQL benchmark, our system achieves 1.41× higher average SLO compliance rate (up to 1.67×) and 1.65× higher throughput (up to 1.75×) compared to vLLM, significantly reducing SLO violations.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search
• Machine Learning: Hardware-aware ML

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Recent advances in leveraging the agentic paradigm of large language models (LLMs) utilization have significantly enhanced Text-to-SQL capabilities, enabling users without specialized database expertise to query data intuitively. However, deploying these agentic LLM-based Text-to-SQL systems in production poses substantial challenges due to their inherently multi-stage workflows, stringent latency constraints, and potentially heterogeneous GPU infrastructure in enterprise environments. Current LLM serving frameworks lack effective mechanisms for handling interdependent inference tasks, dynamic latency variability, and resource heterogeneity, leading to suboptimal performance and frequent service-level objective (SLO) violations. In this paper, we introduce HEXGEN-TEXT2SQL, a novel framework designed explicitly to schedule and execute agentic multi-stage LLM-based Text-to-SQL workflows on heterogeneous GPU clusters that handle multi-tenant end-to-end queries. HEXGEN-TEXT2SQL introduce a hierarchical scheduling approach combining global workload-balanced task dispatching and local adaptive urgency-guided prioritization, guided by a systematic analysis of agentic Text-to-SQL workflows. Additionally, we propose a lightweight simulation-based method for tuning critical scheduling hyperparameters, further enhancing robustness and adaptability. Our extensive evaluation on realistic Text-to-SQL benchmarks demonstrates that HEXGEN-TEXT2SQL significantly outperforms state-of-the-art LLM serving frameworks. Specifically, HEXGEN-TEXT2SQL reduces latency deadlines by up to 1.67$ imes$ (average: 1.41$ imes$) and improves system throughput by up to 1.75$ imes$ (average: 1.65$ imes$) compared to vLLM under diverse, realistic workload conditions. Our code is available at https://github.com/Relaxed-System-Lab/Hexgen-Flow.

Problem

• Optimizing LLM inference scheduling for multi-stage Text-to-SQL workflows
• Addressing latency constraints and GPU heterogeneity in agentic systems
• Improving performance and SLO compliance in LLM-based SQL query systems

Innovation

• Hierarchical scheduling for multi-stage LLM workflows
• Lightweight simulation-based hyperparameter tuning
• Optimized task dispatching on heterogeneous GPU clusters