DiLi: A Lock-Free Asynchronously Distributable Linked List

2025-10-07

AI Summary

To address the challenge of simultaneously achieving scalability and online load balancing in high-throughput distributed databases—where dynamic indexing structures often suffer from coordination overhead or downtime—this paper proposes DiLi, a lock-free linked list supporting asynchronous dynamic distribution. DiLi introduces a novel “conditional lock-freedom” design that guarantees linearizability while enabling zero-downtime partition reconfiguration and load migration. It combines binary search–guided positioning with bounded linear traversal to efficiently support find, insert, and remove operations. On a single node, DiLi matches the performance of state-of-the-art lock-free linked lists; in multi-node deployments, its throughput scales nearly linearly with the number of nodes. Experimental evaluation under high-concurrency workloads demonstrates DiLi’s strong scalability, high availability, and low tail latency—making it particularly suitable for demanding distributed database scenarios.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Distributed Machine Learning & Federated Learning

Application Category

• Security and Privacy: Blockchains and distributed ledgers
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Modern databases use dynamic search structures that store a huge amount of data, and often serve them using multi-threaded algorithms to support the ever-increasing throughput needs. When this throughput need exceeds the capacity of the machine hosting the structure, one either needs to replace the underlying hardware (an option that is typically not viable and introduces a long down time) or make the data structure distributed. Static partitioning of the data structure for distribution is not desirable, as it is prone to uneven load distribution over time, and having to change the partitioning scheme later will require downtime. Since a distributed data structure, inherently, relies on communication support from the network stack and operating systems, we introduce the notion of conditional lock-freedom that extends the notion of lock-free computation with reasonable assumptions about communication between processes. We present DiLi, a conditional lock-free, linearizable, and distributable linked list that can be asynchronously and dynamically (1) partitioned into multiple sublists and (2) load balanced by distributing sublists across multiple machines. DiLi contains primitives for these that also maintain the lock-free property of the underlying search structure that supports find, remove, and insert of a key as the client operations. Searching for an item in DiLi is by a novel traversal that involves a binary search on the partitioning scheme, and then a linear traversal on a limitable number of linked nodes. As a result, we are able to empirically show that DiLi performs as well as the state-of-the-art lock-free concurrent search structures that are based off of a linked list when executed on a single-machine. We also show that the throughput of DiLi scales linearly with the number of machines that host it.

Problem

• Distributes linked lists across machines without downtime
• Maintains lock-free operations during dynamic partitioning
• Enables linear throughput scaling with added machines

Innovation

• Lock-free distributable linked list for dynamic partitioning
• Conditional lock-freedom with asynchronous load balancing
• Binary search partitioning with linear node traversal