AI Query Compilation for Unified and Optimized Execution

2026-08-10

AI Summary

This work addresses the PCIe communication bottleneck and lack of global optimization arising from the decoupled execution of SQL queries and large language model (LLM) inference in AI systems. To overcome these limitations, the paper proposes an end-to-end holistic compilation approach that, for the first time, unifies relational SQL operations and LLM inference into a single tensor computation graph. This unified representation enables global compiler optimizations and automatic sharding, transcending the constraints of conventional execution engines. By integrating with TPU acceleration, the system achieves highly efficient execution. Evaluation on the SemBench Reviews and Movies datasets demonstrates up to a 5.3× reduction in latency and a 9.8× improvement in throughput compared to baseline systems.

Abstract

In this vision paper, we propose a novel architectural paradigm for accelerated AI query execution via a unified compiled execution strategy. By compiling the hybrid AI Query as a whole -- integrating both standard SQL relational constructs and LLM inference layers into a single, unified tensor compute graph -- we completely alleviate PCIe data movement bottlenecks across execution boundaries and enable global compiler optimizations and efficient automatic sharding. We demonstrate the viability of this unified execution paradigm on select and extended AI queries on SemBench Reviews and Movies datasets, achieving up to 5.3x latency speedup and 9.8x throughput speedup on TPUs, and outline a research roadmap of open technical challenges to realize this vision.

Problem

• AI query execution
• PCIe bottleneck
• unified execution
• compiler optimization
• tensor compute graph

Innovation

• AI Query Compilation
• Unified Execution
• Tensor Compute Graph
• Compiler Optimization
• Automatic Sharding