HierarchicalKV: A GPU Hash Table with Cache Semantics for Continuous Online Embedding Storage
This work addresses severe memory inefficiency in conventional GPU hash tables when embedding tables exceed the capacity of a single GPU’s high-bandwidth memory (HBM), as these structures retain all key-value pairs regardless of access patterns. To overcome this limitation, the authors propose HierarchicalKV—the first GPU hash table that treats caching semantics as a first-class operation. It replaces traditional dictionary semantics with a policy-driven eviction mechanism that either updates entries in place or rejects insertions, thereby avoiding costly rehashing and overflow failures. Key innovations include cache-line-aligned buckets, inline score-driven upserts, dynamic dual-bucket selection, three-level concurrency control, and a hierarchical key-value separation architecture. Evaluated on an NVIDIA H100 NVL, HierarchicalKV achieves up to 3.9 billion key-value operations per second, maintains load factors between 0.50 and 1.00 with less than 5% throughput variation, outperforms WarpCore by 1.4×, and surpasses indirect-addressing baselines by 2.6–9.4×, with integration already adopted in multiple open-source recommendation frameworks.