Performance Optimization in Stream Processing Systems: Experiment-Driven Configuration Tuning for Kafka Streams

2026-03-04

AI Summary

This work addresses the inefficiency of manual performance tuning in cloud-native stream processing systems, which heavily relies on expert experience. To automate and accelerate configuration optimization, the authors propose an experiment-driven approach that integrates Latin hypercube sampling, simulated annealing, and hill climbing into a three-stage search strategy. This method is deeply coupled with the Theodolite benchmarking framework to automatically orchestrate experiments on Kubernetes and preemptively terminate underperforming configurations. Evaluated on Kafka Streams, the approach efficiently explores the configuration space and identifies settings that substantially outperform default configurations, achieving up to a 23% improvement in throughput. The study demonstrates a practical and effective pathway toward automated, high-efficiency tuning of stream processing systems in cloud-native environments.

Technology Category

• Search and Optimization: Algorithm Configuration
• Machine Learning: Auto ML and Hyperparameter Tuning
• Data Mining & Knowledge Management: Data Stream Mining

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Configuring stream processing systems for efficient performance, especially in cloud-native deployments, is a challenging and largely manual task. We present an experiment-driven approach for automated configuration optimization that combines three phases: Latin Hypercube Sampling for initial exploration, Simulated Annealing for guided stochastic search, and Hill Climbing for local refinement. The workflow is integrated with the cloud-native Theodolite benchmarking framework, enabling automated experiment orchestration on Kubernetes and early termination of underperforming configurations. In an experimental evaluation with Kafka Streams and a Kubernetes-based cloud testbed, our approach identifies configurations that improve throughput by up to 23% over the default. The results indicate that Latin Hypercube Sampling with early termination and Simulated Annealing are particularly effective in navigating the configuration space, whereas additional fine-tuning via Hill Climbing yields limited benefits.

Problem

• stream processing
• performance optimization
• configuration tuning
• cloud-native
• Kafka Streams

Innovation

• Configuration Optimization
• Stream Processing
• Experiment-Driven Tuning
• Cloud-Native
• Kafka Streams