🤖 AI Summary
This work addresses the high latency in real-time stream processing caused by tight coupling between state I/O and the data path, which blocks the CPU on the critical path. To mitigate this, the authors propose Keyed Prefetching, a mechanism that extracts state-access keys from upstream operators and proactively prefetches the required state, thereby overlapping I/O with computation to hide latency. Complementing this, a Timestamp-Aware Caching strategy is introduced to efficiently manage prefetched and historical states in memory. Together, these techniques significantly reduce end-to-end latency for long-running real-time queries while maintaining high throughput and effectively decoupling state access from data processing.
📝 Abstract
Mission-critical applications often run "forever" and process large data volumes in real time while demanding low latency. To handle the large state of these applications, modern streaming engines rely on key-value stores and store state on local storage or remotely, but accessing such state inflates latency. As today's engines tightly couple the data path with state I/O, a tuple triggers state access only when it reaches a stateful operator, placing I/O on the critical path and stalling the CPU. However, the keys used to access the state are frequently known earlier in the query plan. Building on this insight, we propose Keyed Prefetching, which decouples the data path from state access by extracting future access keys at upstream operators and proactively staging the corresponding state in memory before tuples arrive. This overlaps I/O with ongoing computation and hides the latency of large-state accesses. We pair Keyed Prefetching with Timestamp-Aware Caching, a cache-eviction policy that jointly manages previously accessed and prefetched entries to use memory efficiently. Together, these techniques reduce latency for long-running, real-time queries without sacrificing throughput.