Analysis and Evaluation of Using Microsecond-Latency Memory for In-Memory Indices and Caches in SSD-Based Key-Value Stores

2025-10-14

AI Summary

SSD-based key-value (KV) stores heavily rely on expensive host DRAM for indexing and caching, creating a cost bottleneck. This work investigates migrating these data structures to lower-cost secondary memory with microsecond-scale latency (1–5 μs), but faces the challenge of sustaining high KV throughput under such latency. Method: We propose a user-thread–based software prefetching mechanism that explicitly models and jointly optimizes prefetching and I/O scheduling—without requiring new hardware or protocol modifications—to effectively hide secondary memory latency. Contribution/Results: Using an FPGA-based tunable-latency memory platform, we validate our approach via microbenchmarks and a modified KV store. Results show near-DRAM throughput is maintained even at up to 5 μs memory latency. To our knowledge, this is the first work to demonstrate that microsecond-latency secondary memory can serve as a cost-effective, drop-in DRAM alternative—achieving this without custom hardware or modifications to the storage stack.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Hardware-aware ML

Application Category

• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

When key-value (KV) stores use SSDs for storing a large number of items, oftentimes they also require large in-memory data structures including indices and caches to be traversed to reduce IOs. This paper considers offloading most of such data structures from the costly host DRAM to secondary memory whose latency is in the microsecond range, an order of magnitude longer than those of currently available DIMM-mounted or CXL memory devices. While emerging microsecond-latency memory is likely to cost much less than DRAM, it can significantly slow down SSD-based KV stores if naively employed. This paper analyzes and evaluates the impact of microsecond-level memory latency on the KV operation throughput. Our analysis finds that a well-known latency-hiding technique of software prefetching for long-latency memory from user-level threads is effective. The novelty of our analysis lies in modeling how the interplay between prefetching and IO affects performance, from which we derive an equation that well explains the throughput degradation due to long memory latency. The model tells us that the presence of IO significantly enhances the tolerance to memory latency, leading to a finding that SSD-based KV stores can be made latency-tolerant without devising new techniques for microsecond-latency memory. To confirm this, we design a microbenchmark as well as modify existing SSD-based KV stores so that they issue prefetches from user-level threads, and run them while placing most of in-memory data structures on FPGA-based memory with adjustable microsecond latency. The results demonstrate that their KV operation throughputs can be well explained by our model, and the modified KV stores achieve near-DRAM throughputs for up to a memory latency of 5 microseconds. This suggests the possibility that SSD-based KV stores can use microsecond-latency memory as a cost-effective alternative to the host DRAM.

Problem

• Evaluating microsecond-latency memory for SSD key-value store indices
• Analyzing performance impact of long memory latency on KV operations
• Determining if microsecond-latency memory can replace costly DRAM

Innovation

• Using microsecond-latency memory for in-memory indices
• Applying software prefetching from user-level threads
• Achieving near-DRAM throughput with microsecond-latency memory