Walrus: An Efficient Decentralized Storage Network

2025-05-08

AI Summary

Decentralized blob storage faces a fundamental trade-off among redundancy overhead, recovery efficiency, and security verification: full replication incurs prohibitive storage costs, while conventional erasure coding suffers from inefficient recovery and weak verifiability under high node churn. This paper proposes RedStuff—the first two-dimensional erasure coding protocol supporting asynchronous networks and verifiable storage challenges. It introduces a multi-stage epoch transition mechanism to enable smooth committee rotation and continuous availability, and integrates authenticated data structures—including Merkle B-trees—to ensure data integrity and resilience against malicious clients. Evaluation shows RedStuff achieves only 4.5× storage overhead, reduces recovery bandwidth to O(|blob|/n), and maintains high availability and strong security at scale (1,000 nodes).

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Data Compression
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Data transparency and provenance
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Decentralized storage systems face a fundamental trade-off between replication overhead, recovery efficiency, and security guarantees. Current approaches either rely on full replication, incurring substantial storage costs, or employ trivial erasure coding schemes that struggle with efficient recovery especially under high storage-node churn. We present Walrus, a novel decentralized blob storage system that addresses these limitations through multiple technical innovations. At the core of Walrus is RedStuff, a two-dimensional erasure coding protocol that achieves high security with only 4.5x replication factor, while enabling self-healing recovery that requires bandwidth proportional to only the lost data $(O(|blob|/n)$ versus $O(|blob|)$ in traditional systems). Crucially, RedStuff is the first protocol to support storage challenges in asynchronous networks, preventing adversaries from exploiting network delays to pass verification without actually storing data. Walrus also introduces a novel multi-stage epoch change protocol that efficiently handles storage node churn while maintaining uninterrupted availability during committee transitions. Our system incorporates authenticated data structures to defend against malicious clients and ensures data consistency throughout storage and retrieval processes. Experimental evaluation demonstrates that Walrus achieves practical performance at scale, making it suitable for a wide range of decentralized applications requiring high-integrity, available blob storage with reasonable overhead.

Problem

• Balancing replication overhead, recovery efficiency, and security in decentralized storage
• Improving erasure coding for efficient recovery under high node churn
• Enabling secure storage challenges in asynchronous networks

Innovation

• RedStuff: 2D erasure coding with 4.5x replication
• Self-healing recovery with O(|blob|/n) bandwidth
• Multi-stage epoch change for node churn