Pilotfish: Distributed Execution for Scalable Blockchains

2024-01-29

AI Summary

Blockchain systems under high load rely on over-provisioned validators, leading to resource waste, elevated operational costs, and performance bottlenecks. To address this, we propose Pilotfish—the first scale-out transaction execution engine designed specifically for blockchains. Our approach introduces three key innovations: (1) a distributed execution architecture enabling dynamic validator scheduling across multiple worker nodes for elastic scalability; (2) a versioned queue scheduling algorithm ensuring strong consistency and deterministic execution; and (3) a lightweight partial crash recovery mechanism natively integrated with lazy blockchain architectures. Experimental evaluation under compute-intensive workloads demonstrates near-linear scalability up to eight worker nodes, stable end-to-end latency, significantly improved resource utilization, and high cost-effectiveness with minimal overhead.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Planning, Routing, and Scheduling: Plan Execution and Monitoring

Application Category

• Economics, Online Markets and Human Computation: Economic aspects of blockchain and cryptocurrencies
• Security and Privacy: Blockchains and distributed ledgers
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Scalability is a crucial requirement for modern large-scale systems, enabling elasticity and ensuring responsiveness under varying load. While cloud systems have achieved scalable architectures, blockchain systems remain constrained by the need to over-provision validator machines to handle peak load. This leads to resource inefficiency, poor cost scaling, and limits on performance. To address these challenges, we introduce Pilotfish, the first scale-out transaction execution engine for blockchains. Pilotfish enables validators to scale horizontally by distributing transaction execution across multiple worker machines, allowing elasticity without compromising consistency or determinism. It integrates seamlessly with the lazy blockchain architecture, completing the missing piece of execution elasticity. To achieve this, Pilotfish tackles several key challenges: ensuring scalable and strongly consistent distributed transactions, handling partial crash recovery with lightweight replication, and maintaining concurrency with a novel versioned-queue scheduling algorithm. Our evaluation shows that Pilotfish scales linearly up to at least eight workers per validator for compute-bound workloads, while maintaining low latency. By solving scalable execution, Pilotfish brings blockchains closer to achieving end-to-end elasticity, unlocking new possibilities for efficient and adaptable blockchain systems.

Problem

• Scalability in blockchain systems
• Resource inefficiency in validators
• Elastic transaction execution

Innovation

• Distributed transaction execution engine
• Horizontal scaling with worker machines
• Versioned-queue scheduling algorithm