GPUSparse: GPU-Accelerated Learned Sparse Retrieval with Parallel Inverted Indices

2026-06-24

AI Summary

This work addresses the real-time performance bottleneck of learning-based sparse retrieval models, such as SPLADE, which rely on CPU-based inverted index traversal during inference. To overcome this limitation, the authors propose an efficient GPU-accelerated solution featuring a block-aligned, warp-coalesced parallel inverted index structure, combined with a batched scatter-add scoring algorithm and a fused Triton kernel. A block-wise maximum optimization is further introduced to balance computational efficiency and memory bandwidth utilization. Experimental results on the MS MARCO dataset demonstrate that the proposed method achieves identical retrieval accuracy to exact CPU scoring (MRR@10 = 0.383), while attaining a per-query latency of only 1.27 ms and a throughput of 787 queries per second—yielding a 235× speedup over the Pyserini CPU implementation.

Abstract

Learned sparse retrieval models such as SPLADE achieve retrieval quality competitive with dense models while preserving the interpretability and exact-match advantages of sparse representations. However, inference-time scoring still relies on CPU-bound inverted index traversal algorithms (WAND, Block-Max WAND), creating a fundamental bottleneck for real-time serving at scale. We present GPUSparse, a system for GPU-accelerated exact learned sparse retrieval that introduces: (1) a GPU-parallel inverted index with block-aligned, warp-coalesced posting lists; (2) a batched scatter-add scoring algorithm that processes hundreds of queries simultaneously; and (3) fused Triton kernels with an analysis of the tradeoff between work-efficiency and hardware utilization. On MS MARCO passage ranking (8.8M passages) with real SPLADE embeddings, GPUSparse matches CPU exact scoring to three decimals (MRR@10=0.383, equal to Pyserini SPLADE at this precision; Recall@1000>=0.999 vs. dense matmul, the residual from floating-point tie-breaking) while providing a 235x speedup over Pyserini CPU at 8.8M documents (1.27ms vs. 298ms per query). Compared to Seismic (the fastest CPU sparse retrieval system), which trades 25% recall for speed (R@1000=0.738 vs. 0.983 exact), GPUSparse achieves exact scoring at 787 QPS throughput (batch 500) on the full 8.8M collection, with 1.3ms per query. Our document-parallel kernel reaches 62.6% of H100 peak HBM bandwidth, revealing a fundamental work-efficiency vs. bandwidth-efficiency tradeoff in GPU sparse retrieval. The reformulation of sparse scoring as scatter-add over an inverted index is shared with SPARe's iterative mode; our contribution is its fused-kernel realization, which we measure to be 23-270x faster than a faithful SPARe iterative reimplementation.

Problem

• learned sparse retrieval
• inverted index
• GPU acceleration
• real-time serving
• inference bottleneck

Innovation

• GPU-accelerated retrieval
• learned sparse retrieval
• inverted index
• scatter-add scoring
• fused kernel