Experimental Evaluation of Distributed k-Core Decomposition

2024-06-25

arXiv.org

AI Summary

To address memory and communication bottlenecks in k-core decomposition on large-scale distributed graphs, this paper proposes a decentralized, purely local, and asynchronous distributed algorithm: each vertex iteratively updates its core number solely based on local adjacency information, without requiring a global view or centralized coordination. For the first time, the algorithm is implemented in Go using lightweight goroutines to simulate concurrent computation on million-vertex graphs, and incorporates a message-passing-based local iterative pruning mechanism. Experiments on thousand-node clusters demonstrate that the algorithm preserves decomposition accuracy while achieving near-linear scalability in runtime; total message volume is reduced by 47%, significantly alleviating both communication overhead and memory pressure. This work establishes an efficient and practical new paradigm for real-time k-core decomposition on ultra-large-scale graphs.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Search and Optimization: Distributed Search
• Machine Learning: Graph-based Machine Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Given an undirected graph, the $k$-core is a subgraph in which each node has at least $k$ connections, which is widely used in graph analytics to identify core subgraphs within a larger graph. The sequential $k$-core decomposition algorithm faces limitations due to memory constraints and data graphs can be inherently distributed. A distributed approach is proposed to overcome limitations by allowing each vertex to independently do calculation by only using local information. This paper explores the experimental evaluation of a distributed $k$-core decomposition algorithm. By assuming that each vertex is a client as a single computing unit, we simulate the process using Golang, leveraging its Goroutine and message passing. Due to the fact that the real-world data graphs can be large with millions of vertices, it is expensive to build such a distributed environment with millions of clients if the experiments run in a real-life scenario. Therefore, our experimental simulation can effectively evaluate the running time and message passing for the distributed $k$-core decomposition.

Problem

• Evaluating distributed k-core decomposition algorithm experimentally
• Overcoming memory constraints in large graph analytics
• Simulating distributed computation using Golang for efficiency

Innovation

• Distributed k-core decomposition using local information
• Simulation with Golang Goroutines for message passing
• Experimental evaluation of running time and messaging