AVOCADO: The Streaming Process Mining Challenge

2025-10-19

AI Summary

Evaluating algorithm performance in streaming process mining lacks standardization and struggles to jointly optimize accuracy, latency, and robustness across diverse real-world challenges (e.g., out-of-order events, memory constraints, throughput variability). Method: We propose AVOCADO—the first open-source evaluation framework specifically designed for streaming process mining. It decouples the *conceptual layer* (defining evaluation dimensions and semantics) from the *instantiation layer* (implementing concrete datasets and algorithm interfaces), enabling scalable, incremental, and realistic assessment—including handling event reordering, throughput fluctuations, and memory overhead. Contribution/Results: AVOCADO integrates streaming-specific metrics (e.g., MAE, RMSE, end-to-end latency) and provides a unified benchmark with automated evaluation pipelines. Experimental validation across multiple algorithms and scenarios demonstrates its effectiveness, significantly improving reproducibility and community collaboration, and establishing a methodological foundation for advancing streaming process mining.

Technology Category

• Data Mining & Knowledge Management: Data Stream Mining
• Machine Learning: Time-Series/Data Streams
• Search and Optimization: Evaluation and Analysis

Application Category

• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Streaming process mining deals with the real-time analysis of streaming data. Event streams require algorithms capable of processing data incrementally. To systematically address the complexities of this domain, we propose AVOCADO, a standardized challenge framework that provides clear structural divisions: separating the concept and instantiation layers of challenges in streaming process mining for algorithm evaluation. The AVOCADO evaluates algorithms on streaming-specific metrics like accuracy, Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Processing Latency, and robustness. This initiative seeks to foster innovation and community-driven discussions to advance the field of streaming process mining. We present this framework as a foundation and invite the community to contribute to its evolution by suggesting new challenges, such as integrating metrics for system throughput and memory consumption, and expanding the scope to address real-world stream complexities like out-of-order event arrival.

Problem

• Standardizing algorithm evaluation for streaming process mining challenges
• Assessing streaming algorithms using accuracy and latency metrics
• Addressing real-world stream complexities like out-of-order events

Innovation

• Standardized challenge framework for streaming process mining
• Separates concept and instantiation layers for evaluation
• Evaluates algorithms using streaming-specific performance metrics