FlowLog: Efficient and Extensible Datalog via Incrementality
🤖 AI Summary
Existing Datalog systems struggle to balance efficiency and generality: specialized engines (e.g., Soufflé) achieve high performance but lack flexibility, while database-embedded approaches (e.g., RecStep) offer modularity yet hinder integration of Datalog-specific optimizations. This paper introduces FlowLog, a novel Datalog engine built upon incremental computation. Its key contributions are: (1) an explicit per-rule relational intermediate representation that decouples recursive control from logical execution; (2) preservation of Datalog-aware optimizations at the logical layer while reusing database execution primitives; and (3) novel techniques—structured optimization, lateral information propagation, and recursion-aware Boolean specialization. Implemented atop Differential Dataflow, FlowLog unifies batch and incremental evaluation via semi-naïve evaluation, logical fusion, subplan reuse, and algebraic specialization. Experiments demonstrate that FlowLog significantly outperforms state-of-the-art Datalog engines and modern databases across diverse recursive workloads, achieving high performance, strong scalability, and architectural simplicity.
Application Category
📝 Abstract
Datalog-based languages are regaining popularity as a powerful abstraction for expressing recursive computations in domains such as program analysis and graph processing. However, existing systems often face a trade-off between efficiency and extensibility. Engines like Souffle achieve high efficiency through domain-specific designs, but lack general-purpose flexibility. Others, like RecStep, offer modularity by layering Datalog on traditional databases, but struggle to integrate Datalog-specific optimizations.
This paper bridges this gap by presenting FlowLog, a new Datalog engine that uses an explicit relational IR per-rule to cleanly separate recursive control (e.g., semi-naive execution) from each rule's logical plan. This boundary lets us retain fine-grained, Datalog-aware optimizations at the logical layer, but also reuse off-the-shelf database primitives at execution. At the logical level (i.e. IR), we apply proven SQL optimizations, such as logic fusion and subplan reuse. To address high volatility in recursive workloads, we adopt a robustness-first approach that pairs a structural optimizer (avoiding worst-case joins) with sideways information passing (early filtering). Built atop Differential Dataflow--a mature framework for streaming analytics--FlowLog supports both batch and incremental Datalog and adds novel recursion-aware optimizations called Boolean (or algebraic) specialization. Our evaluation shows that FlowLog outperforms state-of-the-art Datalog engines and modern databases across a broad range of recursive workloads, achieving superior scalability while preserving a simple and extensible architecture.
Problem
Research questions and friction points this paper is trying to address.
Bridging efficiency and extensibility trade-offs in Datalog systems
Separating recursive control from logical plans for optimization flexibility
Addressing high volatility in recursive workloads with robustness-first approach
Innovation
Methods, ideas, or system contributions that make the work stand out.
Explicit relational IR separates recursive control from logical plan
Applies SQL optimizations and sideways information passing techniques
Leverages Differential Dataflow for batch and incremental processing
🔎 Similar Papers
No similar papers found.
