SkyStore: Cost-Optimized Object Storage Across Regions and Clouds

2025-02-28

AI Summary

In multi-cloud environments, cross-cloud object storage faces challenges including API heterogeneity, complex cost modeling, and suboptimal data placement decisions—leading to high operational overhead and increased total cost of ownership (TCO). This paper proposes the first TTL-aware dynamic data placement framework, which (1) virtualizes heterogeneous object/bucket APIs into a unified interface, (2) constructs a fine-grained, multi-cloud cost model incorporating storage, egress, request, and replication expenses, and (3) enables distributed metadata-driven real-time replication and eviction scheduling for cross-cloud, cross-region data layout optimization. Its core innovation lies in jointly modeling access frequency and temporal validity as a dynamic TTL, thereby guiding global cost-minimizing placement decisions. Experimental evaluation demonstrates that, while maintaining equivalent latency and availability, our system reduces total cost by up to 6× compared to state-of-the-art academic baselines and AWS Multi-Region Buckets.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Cloud, edge and content delivery systems for the Web
• Security and Privacy: Data transparency and provenance

Abstract

Modern applications span multiple clouds to reduce costs, avoid vendor lock-in, and leverage low-availability resources in another cloud. However, standard object stores operate within a single cloud, forcing users to manually manage data placement across clouds, i.e., navigate their diverse APIs and handle heterogeneous costs for network and storage. This is often a complex choice: users must either pay to store objects in a remote cloud, or pay to transfer them over the network based on application access patterns and cloud provider cost offerings. To address this, we present SkyStore, a unified object store that addresses cost-optimal data management across regions and clouds. SkyStore introduces a virtual object and bucket API to hide the complexity of interacting with multiple clouds. At its core, SkyStore has a novel TTL-based data placement policy that dynamically replicates and evicts objects according to application access patterns while optimizing for lower cost. Our evaluation shows that across various workloads, SkyStore reduces the overall cost by up to 6x over academic baselines and commercial alternatives like AWS multi-region buckets. SkyStore also has comparable latency, and its availability and fault tolerance are on par with standard cloud offerings. We release the data and code of SkyStore at https://github.com/skyplane-project/skystore.

Problem

• Optimizes data storage costs across multiple cloud regions.
• Simplifies data management with a unified API for multi-cloud environments.
• Dynamically adjusts data placement based on access patterns and costs.

Innovation

• Unified object store for multi-cloud management
• TTL-based dynamic data placement policy
• Virtual API simplifies multi-cloud interactions