Abacus: A Cost-Based Optimizer for Semantic Operator Systems

2025-05-20

AI Summary

This work addresses the challenge of jointly optimizing quality, cost, and latency for LLM-driven semantic operators—such as natural language–to–SQL mapping, filtering, and joining—within globally physicalized execution plans. We propose the first scalable, cost-model–driven optimizer for such systems. Our approach integrates semantic operator abstractions, systematic enumeration of physical execution plans, a cost-based search framework, and a lightweight performance estimation model. Crucially, it achieves accurate operator behavior modeling using only a small number of validation samples and prior performance beliefs, enabling constrained multi-objective optimization. Evaluated on BioDEX, CUAD, and MMQA benchmarks, our optimizer improves quality by 18.7%–39.2% over suboptimal baselines, reduces monetary cost by up to 23.6×, and decreases end-to-end latency by up to 4.2×.

Technology Category

• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

LLMs enable an exciting new class of data processing applications over large collections of unstructured documents. Several new programming frameworks have enabled developers to build these applications by composing them out of semantic operators: a declarative set of AI-powered data transformations with natural language specifications. These include LLM-powered maps, filters, joins, etc. used for document processing tasks such as information extraction, summarization, and more. While systems of semantic operators have achieved strong performance on benchmarks, they can be difficult to optimize. An optimizer for this setting must determine how to physically implement each semantic operator in a way that optimizes the system globally. Existing optimizers are limited in the number of optimizations they can apply, and most (if not all) cannot optimize system quality, cost, or latency subject to constraint(s) on the other dimensions. In this paper we present Abacus, an extensible, cost-based optimizer which searches for the best implementation of a semantic operator system given a (possibly constrained) optimization objective. Abacus estimates operator performance by leveraging a minimal set of validation examples and, if available, prior beliefs about operator performance. We evaluate Abacus on document processing workloads in the biomedical and legal domains (BioDEX; CUAD) and multi-modal question answering (MMQA). We demonstrate that systems optimized by Abacus achieve 18.7%-39.2% better quality and up to 23.6x lower cost and 4.2x lower latency than the next best system.

Problem

• Optimizing semantic operator systems for cost, quality, and latency
• Implementing AI-powered data transformations efficiently
• Overcoming limitations of existing optimizers in document processing

Innovation

• Cost-based optimizer for semantic operator systems
• Leverages validation examples for performance estimation
• Optimizes quality, cost, latency with constraints