Fast Collection Operations from Indexed Stream Fusion

2025-07-08

AI Summary

This paper addresses the challenge of efficient traversal and composition of associative set data structures. We propose Index Stream Fusion: a technique that compiles set operations into stateless index stream sequences, enabling runtime fusion of multi-level joins and filters while eliminating intermediate set allocations and redundant traversals. Unlike prior approaches, our method requires no specialized compiler support or staged compilation, achieving high composability and performance competitive with hand-optimized code. We implement the technique across three languages—Lean (for mechanized verification of functional correctness), Morphic, and Rust—delivering a cross-language library. Empirical evaluation shows a 72% average reduction in memory overhead while preserving high performance. Moreover, we present the first formal verification of the core algorithm in Lean, significantly enhancing the reliability and expressive power of set operation libraries.

Technology Category

• Data Mining & Knowledge Management: Data Stream Mining
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

We present a system of efficient methods for traversing and combining associative collection data structures. A distinguishing feature of the system is that, like traditional sequential iterator libraries, it does not require specialized compiler infrastructure or staged compilation for efficiency and composability. By using a representation based on indexed streams, the library can express complex joins over input collections while using no intermediate allocations. We implement the library for the Lean, Morphic, and Rust programming languages and provide a mechanized proof of functional correctness in Lean.

Problem

• Efficient traversal of associative collections
• Combining collections without intermediate allocations
• Functional correctness proof for Lean, Morphic, Rust

Innovation

• Uses indexed streams for efficient traversal
• Requires no specialized compiler infrastructure
• Eliminates intermediate allocations in joins