🤖 AI Summary
To address the challenges of large-scale vector graphs exceeding single-GPU memory capacity—causing frequent GPU stalls, low retrieval throughput, and high latency in conventional CPU-GPU architectures—this paper proposes a pipelined approximate nearest neighbor (ANN) search system for GPU clusters. Our method introduces an asynchronous data transfer mechanism leveraging GPUDirect Async to overlap computation with communication, and establishes a distributed vector index management framework coupled with pipelined parallel computation to support ultra-large-batch queries. Experimental results demonstrate that, while maintaining high recall (>95%), our system achieves a 3.2× throughput improvement and reduces end-to-end latency by 67%, significantly outperforming state-of-the-art CPU-GPU and single-GPU baselines.
📝 Abstract
Vector similarity search has become a critical component in AI-driven applications such as large language models (LLMs). To achieve high recall and low latency, GPUs are utilized to exploit massive parallelism for faster query processing. However, as the number of vectors continues to grow, the graph size quickly exceeds the memory capacity of a single GPU, making it infeasible to store and process the entire index on a single GPU. Recent work uses CPU-GPU architectures to keep vectors in CPU memory or SSDs, but the loading step stalls GPU computation. We present Fantasy, an efficient system that pipelines vector search and data transfer in a GPU cluster with GPUDirect Async. Fantasy overlaps computation and network communication to significantly improve search throughput for large graphs and deliver large query batch sizes.