🤖 AI Summary
This work addresses the performance limitations of modern observability platforms under high-concurrency, compute-intensive queries, where repeated filtering over high-frequency data incurs significant overhead. The authors propose FluxSieve, the first unified data plane architecture that integrates pull-based analytics and push-based stream processing by embedding a lightweight streaming precomputation and filtering layer directly into the ingestion path. Key innovations include a scalable mechanism supporting multi-pattern concurrent matching and dynamic rule updates, along with seamless integration with Apache Pinot and DuckDB. Experimental results demonstrate that FluxSieve achieves orders-of-magnitude performance improvements across diverse systems and query workloads, while introducing only negligible storage overhead and minimal computational burden.
📝 Abstract
Despite many advances in query optimization, indexing techniques, and data storage, modern data platforms still face difficulties in delivering robust query performance under high concurrency and computationally intensive queries. This challenge is particularly pronounced in large-scale observability platforms handling high-volume, high-velocity data records. For instance, recurrent, expensive filtering queries at query time impose substantial computational and storage overheads in the analytical data plane. In this paper, we propose FluxSieve, a unified architecture that reconciles traditional pull-based query processing with push-based stream processing by embedding a lightweight in-stream precomputation and filtering layer directly into the data ingestion path. This avoids the complexity and operational burden of running queries in dedicated stream processing frameworks. Concretely, this work (i) introduces a foundational architecture that unifies streaming and analytical data planes via in-stream filtering and records enrichment, (ii) designs a scalable multi-pattern matching mechanism that supports concurrent evaluation and on-the-fly updates of filtering rules with minimal per-record overhead, (iii) demonstrates how to integrate this ingestion-time processing with two open-source analytical systems -- Apache Pinot as a Real-Time Online Analytical Processing (RTOLAP) engine and DuckDB as an embedded analytical database, and (iv) performs comprehensive experimental evaluation of our approach. Our evaluation across different systems, query types, and performance metrics shows up to orders-of-magnitude improvements in query performance at the cost of negligible additional storage and very low computational overhead.