TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval

2025-02-28

AI Summary

This work addresses the high end-to-end latency of RAG systems under GPU memory constraints. We propose Lookahead Retrieval—a novel retrieval strategy that dynamically prefetches high-potential documents by leveraging query similarity modeling and the intrinsic structure of IVF (Inverted File) indexes. To maximize hardware utilization, we introduce CPU-GPU cooperative scheduling that deeply overlaps retrieval data movement with LLM generation via fine-grained pipelining. Technically, our approach integrates IVF-based acceleration, decoupling of computation and data transfer, modular RAG architecture design, and precise pipeline control. Evaluated in realistic low-GPU-memory settings, our method achieves an average 1.72× reduction in end-to-end inference latency and significantly lowers peak GPU memory consumption—enabling, for the first time, millisecond-level responsive, lightweight RAG deployment in real-time scenarios.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Image and Video Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Retrieval-augmented generation (RAG) extends large language models (LLMs) with external data sources to enhance factual correctness and domain coverage. Modern RAG pipelines rely on large datastores, leading to system challenges in latency-sensitive deployments, especially when limited GPU memory is available. To address these challenges, we propose TeleRAG, an efficient inference system that reduces RAG latency with minimal GPU memory requirements. The core innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that anticipates required data and transfers it from CPU to GPU in parallel with LLM generation. By leveraging the modularity of RAG pipelines, the inverted file index (IVF) search algorithm and similarities between queries, TeleRAG optimally overlaps data movement and computation. Experimental results show that TeleRAG reduces end-to-end RAG inference latency by up to 1.72x on average compared to state-of-the-art systems, enabling faster, more memory-efficient deployments of advanced RAG applications.

Problem

• Reduces RAG latency with minimal GPU memory
• Enhances factual correctness and domain coverage
• Optimizes data movement and computation overlap

Innovation

• Lookahead retrieval for prefetching data
• Overlaps data movement and computation efficiently
• Reduces RAG latency with minimal GPU memory