Budget-Constrained Online Retrieval-Augmented Generation: The Chunk-as-a-Service Model

2026-04-28

AI Summary

This study addresses the lack of transparency and cost inefficiency in existing Retrieval-Augmented Generation-as-a-Service (RaaS) billing models, which fail to account for the relevance and actual utilization of retrieved chunks. To overcome these limitations, the authors propose Chunk-as-a-Service (CaaS), a novel framework featuring a transparent, chunk-level billing mechanism, along with an online Utility-Cost Optimal Selection Algorithm (UCOSA) that dynamically selects high-value chunks under budget constraints to enhance large language model outputs. Experimental results demonstrate that UCOSA improves performance by approximately 52% over random selection and achieves 75% of the offline optimal performance. Furthermore, two CaaS variants—LB-CaaS and OB-CaaS—exhibit performance-to-budget ratios that surpass conventional RaaS by 140% and 86%, respectively.

Abstract

Large Language Models (LLMs) have revolutionized the field of natural language processing. However, they exhibit some limitations, including a lack of reliability and transparency: they may hallucinate and fail to provide sources that support the generated output. Retrieval-Augmented Generation (RAG) was introduced to address such limitations in LLMs. One popular implementation, RAG-as-a-Service (RaaS), has shortcomings that hinder its adoption and accessibility. For instance, RaaS pricing is based on the number of submitted prompts, without considering whether the prompts are enriched by relevant chunks, i.e., text segments retrieved from a vector database, or the quality of the utilized chunks (i.e., their degree of relevance). This results in an opaque and less cost-effective payment model. We propose Chunk-as-a-Service (CaaS) as a transparent and cost-effective alternative. CaaS includes two variants: Open-Budget CaaS (OB-CaaS) and Limited-Budget CaaS (LB-CaaS), which is enabled by our ``Utility-Cost Online Selection Algorithm (UCOSA)''. UCOSA further extends the cost-effectiveness and the accessibility of the OB-CaaS variant by enriching, in an online manner, a subset of the submitted prompts based on budget constraints and utility-cost tradeoff. Our experiments demonstrate the efficacy of the proposed UCOSA compared to both offline and relevance-greedy selection baselines. In terms of the performance metric-the number of enriched prompts (NEP) multiplied by the Average Relevance (AR)-UCOSA outperforms random selection by approximately 52% and achieves around 75% of the performance of offline selection methods. Additionally, in terms of budget utilization, LB-CaaS and OB-CaaS achieve higher performance-to-budget ratios of 140% and 86%, respectively, compared to RaaS, indicating their superior efficiency.

Problem

• Retrieval-Augmented Generation
• Budget-Constrained
• Cost-Effectiveness
• Chunk Selection
• Online Retrieval

Innovation

• Chunk-as-a-Service
• Retrieval-Augmented Generation
• online selection algorithm
• budget-constrained optimization
• utility-cost tradeoff