Accessible and Portable LLM Inference by Compiling Computational Graphs into SQL

2025-02-05

AI Summary

Deploying large language models (LLMs) on edge and resource-constrained devices remains challenging due to reliance on specialized hardware (e.g., GPUs) and proprietary inference frameworks. Method: This paper introduces SQL-LLM—a novel approach that automatically compiles LLM inference computation graphs into standard SQL, enabling end-to-end Transformer inference natively within general-purpose relational databases. Contribution/Results: SQL-LLM is the first to express neural operators—including matrix multiplication and attention—completely in SQL; it pioneers in-database KV caching and disk-aware inference state management; and it formalizes attention via relational algebra. Experiments demonstrate that, under CPU-only and memory-constrained settings, SQL-LLM achieves up to 30× higher throughput than state-of-the-art CPU-based LLM frameworks for Llama3 inference—without any GPU dependency. It supports mainstream databases including PostgreSQL and SQLite, significantly improving accessibility, portability, and hardware agnosticism of LLM deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Serving large language models (LLMs) often demands specialized hardware, dedicated frameworks, and substantial development efforts, which restrict their accessibility, especially for edge devices and organizations with limited technical resources. We propose a novel compiler that translates LLM inference graphs into SQL queries, enabling relational databases, one of the most widely used and mature software systems globally, to serve as the runtime. By mapping neural operators such as matrix multiplication and attention into relational primitives like joins and aggregations, our approach leverages database capabilities, including disk-based data management and native caching. Supporting key transformer components, such as attention mechanisms and key-value caching, our system generates SQL pipelines for end-to-end LLM inference. Using the Llama3 family as a case study, we demonstrate up to 30x speedup in token generation for memory-constrained scenarios comparable to competitive CPU-based frameworks. Our work offers an accessible, portable, and efficient solution, facilitating the serving of LLMs across diverse deployment environments.

Problem

• Compiles LLM inference graphs into SQL
• Enables relational databases for LLM runtime
• Facilitates accessible LLM serving on edge devices

Innovation

• Compiles LLM graphs to SQL
• Uses relational database runtime
• Maps neural operators to SQL