Revisiting the Design of In-Memory Dynamic Graph Storage

2025-02-10

Proceedings of the ACM on Management of Data

AI Summary

This paper addresses performance bottlenecks of in-memory dynamic graph storage (DGS) under high-concurrency read/write workloads, identifying three fundamental issues: (1) excessive memory redundancy—3.3× to 8.9× higher than CSR; (2) poor cache efficiency due to neglect of modern memory access patterns; and (3) severe contention and versioning overhead on high-degree vertices caused by fine-grained concurrency control. To systematically analyze these challenges, the authors propose a unified DGS abstraction model and a configurable, multi-dimensional benchmarking framework—the first to quantitatively evaluate trade-offs among throughput, latency, memory footprint, and cache behavior across mainstream DGS designs. Empirical results demonstrate that fine-grained versioning is fundamentally unsuitable for highly concurrent dynamic graph workloads. The study provides both theoretical foundations and practical guidance for designing next-generation DGS architectures that are low-overhead, cache-friendly, and scalable.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

The effectiveness of in-memory dynamic graph storage (DGS) for supporting concurrent graph read and write queries is crucial for real-time graph analytics and updates. Various methods have been proposed, for example, LLAMA, Aspen, LiveGraph, Teseo, and Sortledton. These approaches differ significantly in their support for read and write operations, space overhead, and concurrency control. However, there has been no systematic study to explore the trade-offs among these dimensions. In this paper, we evaluate the effectiveness of individual techniques and identify the performance factors affecting these storage methods by proposing a common abstraction for DGS design and implementing a generic test framework based on this abstraction. Our findings highlight several key insights: 1) Existing DGS methods exhibit substantial space overhead. For example, Aspen consumes 3.3-10.8x more memory than CSR, while the optimal fine-grained methods consume 4.1-8.9x more memory than CSR, indicating a significant memory overhead. 2) Existing methods often overlook memory access impact of modern architectures, leading to performance degradation compared to continuous storage methods. 3) Fine-grained concurrency control methods, in particular, suffer from severe efficiency and space issues due to maintaining versions and performing checks for each neighbor. These methods also experience significant contention on high-degree vertices. Our systematic study reveals these performance bottlenecks and outlines future directions to improve DGS for real-time graph analytics.

Problem

• Evaluates in-memory dynamic graph storage effectiveness
• Identifies performance factors in graph storage methods
• Highlights space overhead and concurrency control issues

Innovation

• Common abstraction for DGS design
• Generic test framework implementation
• Evaluation of concurrency control methods