Designing Parallel Algorithms for Community Detection using Arachne

2025-07-08

AI Summary

Existing Python graph libraries (e.g., NetworkX, igraph) suffer from limited parallelism for large-scale graph community detection. To address this, we design and implement highly scalable parallel label propagation and Louvain algorithms atop the Arachne framework. Our approach innovatively integrates fine-grained task scheduling, memory locality optimization, and multi-threaded load balancing into Arachne, substantially improving concurrency efficiency. Experimental evaluation on standard benchmark graphs demonstrates speedups of up to 710×, 75×, and 12× over NetworkX, igraph, and NetworKit, respectively. Moreover, the proposed methods exhibit strong scalability with increasing numbers of threads and graph sizes. This work delivers an efficient, user-friendly parallel solution for community detection in large-scale graphs.

Technology Category

• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Machine Learning: Graph-based Machine Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Social Networks and Social Media: Social mining and social search on the Web

Abstract

The rise of graph data in various fields calls for efficient and scalable community detection algorithms. In this paper, we present parallel implementations of two widely used algorithms: Label Propagation and Louvain, specifically designed to leverage the capabilities of Arachne which is a Python-accessible, open-source framework for large-scale graph analysis. Our implementations achieve substantial speedups over existing Python-based tools like NetworkX and igraph, which lack efficient parallelization, and are competitive with parallel frameworks such as NetworKit. Experimental results show that Arachne-based methods outperform these baselines, achieving speedups of up to 710x over NetworkX, 75x over igraph, and 12x over NetworKit. Additionally, we analyze the scalability of our implementation under varying thread counts, demonstrating how different phases contribute to overall performance gains of the parallel Louvain algorithm. Arachne, including our community detection implementation, is open-source and available at https://github.com/Bears-R-Us/arkouda-njit .

Problem

• Developing parallel community detection algorithms for large graphs
• Enhancing speed over Python tools like NetworkX and igraph
• Analyzing scalability of parallel Louvain algorithm performance

Innovation

• Parallel Label Propagation and Louvain algorithms
• Leveraging Arachne for scalable graph analysis
• Achieving 710x speedup over NetworkX