syftr: Pareto-Optimal Generative AI

2025-05-26

AI Summary

RAG pipelines face a fundamental trade-off among accuracy, latency, and cost in dynamic or private-data settings, exacerbated by tight coupling and hyperparameter sensitivity across modules (e.g., embedding models, chunkers, validators, query rewriters). This work introduces the first multi-objective Bayesian optimization framework tailored for RAG configuration. It innovatively integrates Pareto-front modeling to balance competing objectives, early-stopping–driven efficient search to reduce evaluation overhead, and an agent-based modular orchestration mechanism enabling end-to-end automated pipeline configuration. Evaluated on multiple standard RAG benchmarks, our method achieves state-of-the-art accuracy while reducing average deployment cost by approximately 9×. The framework significantly enhances the cost-efficiency and practical deployability of generative AI systems without compromising retrieval or generation quality.

Technology Category

• Search and Optimization: Algorithm Configuration
• Planning, Routing, and Scheduling: Planning with Language Models
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Retrieval-Augmented Generation (RAG) pipelines are central to applying large language models (LLMs) to proprietary or dynamic data. However, building effective RAG flows is complex, requiring careful selection among vector databases, embedding models, text splitters, retrievers, and synthesizing LLMs. The challenge deepens with the rise of agentic paradigms. Modules like verifiers, rewriters, and rerankers-each with intricate hyperparameter dependencies have to be carefully tuned. Balancing tradeoffs between latency, accuracy, and cost becomes increasingly difficult in performance-sensitive applications. We introduce syftr, a framework that performs efficient multi-objective search over a broad space of agentic and non-agentic RAG configurations. Using Bayesian Optimization, syftr discovers Pareto-optimal flows that jointly optimize task accuracy and cost. A novel early-stopping mechanism further improves efficiency by pruning clearly suboptimal candidates. Across multiple RAG benchmarks, syftr finds flows which are on average approximately 9 times cheaper while preserving most of the accuracy of the most accurate flows on the Pareto-frontier. Furthermore, syftr's ability to design and optimize allows integrating new modules, making it even easier and faster to realize high-performing generative AI pipelines.

Problem

• Optimizing RAG pipelines for accuracy and cost efficiency
• Managing complex dependencies in agentic RAG configurations
• Balancing tradeoffs between latency, accuracy, and cost

Innovation

• Multi-objective search for RAG configurations
• Bayesian Optimization for Pareto-optimal flows
• Early-stopping mechanism prunes suboptimal candidates