Hash Table Design for RDMA:Challenges and Opportunities

2026-06-22

AI Summary

This work addresses the challenges of deploying hash tables in RDMA-based remote memory under IoT big data scenarios, where limited local memory necessitates offloading to disaggregated memory systems. The key obstacles include frequent remote accesses, ineffective concurrency control, and constrained RDMA resources. To tackle these issues, the study presents the first holistic design that jointly optimizes hash collision resolution, concurrency mechanisms, and RDMA hardware characteristics. By leveraging one-sided RDMA operations, lightweight concurrency control, and careful structuring of NIC resources—including queues, atomic operations, and memory registration—the proposed paradigm significantly improves latency, throughput, and scalability of distributed key-value systems in memory-disaggregated architectures.

Abstract

Hash tables complete the insertion, lookup, and deletion of a single key in constant time on average, and they are widely used in databases, key-value stores, and network systems. In the Internet of Things (IoT), the number of devices and the volume of sensed data keep growing, so the hash tables that store or index these data consume more and more memory. When a single server runs out of memory, the system can place part of the data in the memory of other nodes. One-sided operations in Remote Direct Memory Access (RDMA) let one machine read and write the memory of another machine directly, with low latency and high bandwidth, and are therefore widely used to build disaggregated memory systems. Deploying a hash table on RDMA-based remote memory exploits the memory of other nodes and thus relieves the capacity limit of a single server. However, this deployment raises three problems. First, one logical hash-table access may translate into one or more remote network accesses, and collision handling and probing further increase the number of RDMA requests. Second, because the remote CPU is bypassed, traditional concurrency control that relies on remote threads no longer applies directly. Third, the limited resources of RDMA network interface cards, such as queues, caches, memory registration, and atomic operations, impose new constraints on hash table structures. This paper focuses on hash table design for RDMA. We review existing work, distill the key challenges, and discuss promising optimization directions and coping strategies, aiming to provide a reference for designing remote hash tables in IoT big-data scenarios.

Problem

• Hash Table
• RDMA
• Concurrency Control
• Memory Disaggregation
• Network Access

Innovation

• RDMA
• hash table
• disaggregated memory
• concurrency control
• one-sided operations