Host-guided data placement: whose job is it anyway?

2025-01-01

AI Summary

Emerging SSD interface standards exacerbate software incompatibility, suboptimal data placement, system instability, and weak cross-platform support. To address these challenges, this paper proposes Reshim—the first fully userspace shim layer for SSDs. Reshim introduces the novel design paradigm of “isolated data placement logic,” enabling dynamic, interface- and application-agnostic rule deployment via host-device co-designed affinity-awareness and data-lifecycle-driven policies—without modifying the OS or applications. It supports zero-intrusion integration with RocksDB, MongoDB, and CacheLib. Evaluation demonstrates 2–6× higher write throughput, up to 6× lower tail latency, and significant write amplification reduction. Reshim matches ZenFS in overall performance while achieving lower latency, greater placement policy flexibility, and broader generality across storage interfaces and workloads.

Technology Category

• Search and Optimization: Distributed Search
• Humans and AI: Intelligent User Interfaces
• Machine Learning: Hardware-aware ML

Application Category

• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Security and Privacy: Data transparency and provenance

Abstract

The increasing demand for SSDs coupled with scaling difficulties have left manufacturers scrambling for newer SSD interfaces which promise better performance and durability. While these interfaces reduce the rigidity of traditional abstractions, they require application or system-level changes that can impact the stability, security, and portability of systems. To make matters worse, such changes are rendered futile with introduction of next-generation interfaces. Further, there is little guidance on data placement and hardware specifics are often abstracted from the application layer. It is no surprise therefore that such interfaces have seen limited adoption, leaving behind a graveyard of experimental interfaces ranging from open-channel SSDs to zoned namespaces. In this paper, we show how shim layers can to shield systems from changing hardware interfaces while benefiting from them. We present Reshim, an all-userspace shim layer that performs affinity and lifetime based data placement with no change to the operating system or the application. We demonstrate Reshim's ease of adoption with host-device coordination for three widely-used data-intensive systems: RocksDB, MongoDB, and CacheLib. With Reshim, these systems see 2-6 times highe write throughput, up to 6 times lower latency, and reduced write amplification compared to filesystems like F2FS. Reshim performs on par with application-specific backends like ZenFS while offering more generality, lower latency, and richer data placement. With Reshim we demonstrate the value of isolating the complexity of the placement logic, allowing easy deployment of dynamic placement rules across several applications and storage interfaces.

Problem

• SSD interface compatibility
• data placement efficiency
• system stability and security

Innovation

• Reshim
• Dynamic Data Placement
• Interface Agnosticism