OpRAG: A Resource-Deterministic Runtime for GPU-Backed Multi-Stage RAG Workflows

2026-08-08

AI Summary

Existing RAG systems struggle to efficiently support multi-stage workflows due to sequential execution, scheduling fragmentation, and CPU-GPU pipeline stalls. This work proposes a resource-deterministic distributed runtime that, for the first time, models each RAG stage as a resource-aware first-class operator and constructs a communication-aware execution graph. Deterministic scheduling and low-overhead execution are achieved through a combination of an Arrow-based zero-copy data plane, persistent workers, bounded queues, CPU-based tokenization prefetching, and overlapping retrieval with generation. Evaluated on Llama3-8B and Mistral-7B, the approach reduces end-to-end latency by 17.77% and 17.48%, respectively, while cutting hybrid retrieval-generation latency by 59.20–59.62% and 52.48–53.55%, all while maintaining perfect Recall@5 at 100%.

Abstract

Agentic retrieval-augmented generation (RAG) systems combine preprocessing, embedding, retrieval, memory access, context construction, generation, and vector-index updates. Although LLM decoding is GPU-bound, the surrounding orchestration layer can still limit end-to-end performance through serialization overhead, fragmented scheduling, inefficient batching, and CPU--GPU pipeline stalls. Existing frameworks provide flexible control flow, while distributed runtimes provide scalable task parallelism, but neither exposes RAG stages as resource-aware operators with deterministic execution semantics. We present OpRAG, a resource-deterministic distributed runtime for GPU-backed multi-stage RAG workflows. OpRAG models embedding, retrieval, reasoning, memory, and upsert as first-class operators and lowers them into communication-aware execution graphs. It combines an Arrow zero-copy data plane, persistent workers, bounded queues, CPU tokenizer prefetching, batched GPU embedding, and overlapped retrieval/generation execution to reduce non-model overhead around LLM inference. We evaluate OpRAG using Llama3-8B and Mistral-7B with FlashAttention~2, BF16 execution, and 32K RAG chunks. In end-to-end GPU pipeline experiments, OpRAG improves over the nearest competitor by 16.16% for Llama3-8B and 15.66% for Mistral-7B, and over RayScalableRAG by 20.57% and 20.71%, respectively. Against LangChain, LangGraph, CrewAI, and AutoGen, OpRAG is 17.77% and 17.48% faster than the best framework baseline. In Higress-style query serving, OpRAG reduces hybrid retrieval latency by 59.20--59.62% and generation-scenario latency by 52.48--53.55%, while preserving 100% Recall@5. These results show that optimizing the distributed orchestration layer can substantially improve GPU-backed multi-stage RAG without modifying the LLM decoding kernel.

Problem

• RAG
• GPU
• distributed runtime
• resource-deterministic
• orchestration overhead

Innovation

• resource-deterministic runtime
• multi-stage RAG
• GPU-backed orchestration
• communication-aware execution
• zero-copy data plane