Poseidon: A OneGraph Engine

2025-10-13

AI Summary

This paper addresses the challenge of unifying heterogeneous workloads—real-time transaction processing, dynamic graph top-k search, and holistic graph analytics—in graph databases. To this end, we propose Poseidon, the core engine of Neptune Analytics. Poseidon integrates a lock-free adjacency list, block-partitioned heap storage, lightweight secondary indexing, and a statistics-driven cost model to simultaneously deliver high-throughput transaction processing and low-latency dynamic graph analytics. It extends openCypher with enhanced semantics—including graph pattern matching, variable-length path traversal, aggregation, and inline algorithm invocation—while rigorously distinguishing global IRIs from local identifiers in OneGraph. Experimental results demonstrate a bulk ingestion throughput of 10 million property values per second, sub-20 ms latency for simple transactions, and robust support for real-time applications such as fraud detection.

Technology Category

• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Search and Optimization: Distributed Search
• Machine Learning: Graph-based Machine Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

We present the Poseidon engine behind the Neptune Analytics graph database service. Customers interact with Poseidon using the declarative openCypher query language, which enables requests that seamlessly combine traditional querying paradigms (such as graph pattern matching, variable length paths, aggregation) with algorithm invocations and has been syntactically extended to facilitate OneGraph interoperability, such as the disambiguation between globally unique IRIs (as exposed via RDF) vs. local identifiers (as encountered in LPG data). Poseidon supports a broad range of graph workloads, from simple transactions, to top-k beam search algorithms on dynamic graphs, to whole graph analytics requiring multiple full passes over the data. For example, real-time fraud detection, like many other use cases, needs to reflect current committed state of the dynamic graph. If a users cell phone is compromised, then all newer actions by that user become immediately suspect. To address such dynamic graph use cases, Poseidon combines state-of-the-art transaction processing with novel graph data indexing, including lock-free maintenance of adjacency lists, secondary succinct indices, partitioned heaps for data tuple storage with uniform placement, and innovative statistics for cost-based query optimization. The Poseidon engine uses a logical log for durability, enabling rapid evolution of in-memory data structures. Bulk data loads achieve more than 10 million property values per second on many data sets while simple transactions can execute in under 20ms against the storage engine.

Problem

• Handling dynamic graph workloads with real-time fraud detection
• Integrating openCypher queries with algorithm invocations and interoperability
• Combining transaction processing with novel indexing for graph analytics

Innovation

• Lock-free adjacency list maintenance for dynamic graphs
• Secondary succinct indices for efficient data access
• Partitioned heaps with uniform placement for tuple storage