Approximation-First Timeseries Monitoring Query At Scale

2025-05-15

AI Summary

To address redundant data scanning and high query latency caused by overlapping windows in time-series monitoring systems, this paper proposes PromSketch—the first “approximation-first” intermediate caching framework. PromSketch innovatively integrates Count-Min Sketch with sliding-window approximate aggregation to construct a pluggable caching layer, augmented by a rule-aware cache eviction policy and a lightweight metadata index. It achieves efficient coverage for 70% of prevalent time-series aggregation queries, with average approximation error ≤5%. Experimental evaluation demonstrates that, compared to Prometheus, PromSketch reduces query latency and computational cost by two orders of magnitude; relative to VictoriaMetrics, it cuts cost by over 4×. This work establishes the first systematic framework enabling low-overhead, high-coverage approximate query processing for windowed time-series monitoring workloads.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Machine Learning: Time-Series/Data Streams
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Web Mining and Content Analysis: Topic discovery and tracking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Timeseries monitoring systems such as Prometheus play a crucial role in gaining observability of the underlying system components. These systems collect timeseries metrics from various system components and perform monitoring queries over periodic window-based aggregations (i.e., rule queries). However, despite wide adoption, the operational costs and query latency of rule queries remain high. In this paper, we identify major bottlenecks associated with repeated data scans and query computations concerning window overlaps in rule queries, and present PromSketch, an approximation-first query framework as intermediate caches for monitoring systems. It enables low operational costs and query latency, by combining approximate window-based query frameworks and sketch-based precomputation. PromSketch is implemented as a standalone module that can be integrated into Prometheus and VictoriaMetrics, covering 70% of Prometheus' aggregation over time queries. Our evaluation shows that PromSketch achieves up to a two orders of magnitude reduction in query latency over Prometheus and VictoriaMetrics, while lowering operational dollar costs of query processing by two orders of magnitude compared to Prometheus and by at least 4x compared to VictoriaMetrics with at most 5% average errors across statistics. The source code has been made available at https://github.com/Froot-NetSys/promsketch.

Problem

• Reduces high operational costs in timeseries monitoring queries
• Minimizes query latency for window-based aggregations
• Addresses repeated data scans and window overlap issues

Innovation

• Approximation-first query framework for monitoring
• Sketch-based precomputation to reduce latency
• Standalone module compatible with Prometheus and VictoriaMetrics