[Technical Report] ArceKV: Towards Workload-driven LSM-compactions for Key-Value Store Under Dynamic Workloads

2025-08-05

AI Summary

Existing LSM-tree key-value stores are optimized for static read-write ratios and thus struggle to adapt to dynamic workloads, leading to degraded compaction efficiency and substantial overhead from policy switching. This paper proposes ElasticLSM, an adaptive LSM-tree architecture, and Arce, a lightweight decision engine, implemented atop RocksDB as ArceKV. ArceKV breaks the conventional constraint of fixed operational parameter spaces by enabling runtime elastic compaction scheduling and dynamic write-stalling control. It further introduces a workload-aware, online reinforcement learning decision mechanism that achieves low-overhead, continuous self-adaptation. Experimental evaluation across diverse dynamic workloads demonstrates that ArceKV delivers an average performance improvement of approximately 3× over state-of-the-art compaction strategies, significantly outperforming them in throughput, latency, and resource efficiency.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Data Compression

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Key-value stores underpin a wide range of applications due to their simplicity and efficiency. Log-Structured Merge Trees (LSM-trees) dominate as their underlying structure, excelling at handling rapidly growing data. Recent research has focused on optimizing LSM-tree performance under static workloads with fixed read-write ratios. However, real-world workloads are highly dynamic, and existing workload-aware approaches often struggle to sustain optimal performance or incur substantial transition overhead when workload patterns shift. To address this, we propose ElasticLSM, which removes traditional LSM-tree structural constraints to allow more flexible management actions (i.e., compactions and write stalls) creating greater opportunities for continuous performance optimization. We further design Arce, a lightweight compaction decision engine that guides ElasticLSM in selecting the optimal action from its expanded action space. Building on these components, we implement ArceKV, a full-fledged key-value store atop RocksDB. Extensive evaluations demonstrate that ArceKV outperforms state-of-the-art compaction strategies across diverse workloads, delivering around 3x faster performance in dynamic scenarios.

Problem

• Optimizing LSM-tree performance under dynamic workloads
• Reducing transition overhead during workload pattern shifts
• Enhancing compaction strategies for key-value stores

Innovation

• ElasticLSM removes LSM-tree structural constraints
• Arce engine optimizes compaction decisions dynamically
• ArceKV enhances performance in dynamic workloads