On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks

2025-04-29

AI Summary

This work investigates the capability of large language models (LLMs) to understand and reason about semantic aspects in semantic-aware process mining tasks—such as semantic-driven process discovery and anomaly detection. We introduce the first benchmark dataset covering five semantic-sensitive task categories, enabling unified evaluation of both in-context learning and supervised fine-tuning paradigms. Experiments reveal that prompt engineering alone fails to adequately elicit LLMs’ capacity for modeling process semantics; in contrast, supervised fine-tuning on process logs augmented with semantic annotations substantially improves performance across diverse industries and task types, consistently outperforming zero-shot and few-shot baselines. Our key contributions are: (1) the first LLM evaluation benchmark tailored for semantic-aware process mining; (2) empirical evidence demonstrating that supervised fine-tuning critically enhances LLMs’ semantic reasoning over process data; and (3) validation of LLMs’ practical utility in real-world process analysis scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Data Mining & Knowledge Management: Semantic Web

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large language models (LLMs) have shown to be valuable tools for tackling process mining tasks. Existing studies report on their capability to support various data-driven process analyses and even, to some extent, that they are able to reason about how processes work. This reasoning ability suggests that there is potential for LLMs to tackle semantics-aware process mining tasks, which are tasks that rely on an understanding of the meaning of activities and their relationships. Examples of these include process discovery, where the meaning of activities can indicate their dependency, whereas in anomaly detection the meaning can be used to recognize process behavior that is abnormal. In this paper, we systematically explore the capabilities of LLMs for such tasks. Unlike prior work, which largely evaluates LLMs in their default state, we investigate their utility through both in-context learning and supervised fine-tuning. Concretely, we define five process mining tasks requiring semantic understanding and provide extensive benchmarking datasets for evaluation. Our experiments reveal that while LLMs struggle with challenging process mining tasks when used out of the box or with minimal in-context examples, they achieve strong performance when fine-tuned for these tasks across a broad range of process types and industries.

Problem

• Exploring LLMs' potential for semantics-aware process mining tasks
• Assessing LLMs' ability to understand activity meanings and relationships
• Evaluating fine-tuned LLMs on diverse process mining benchmarks

Innovation

• Utilizing LLMs for semantics-aware process mining tasks
• Exploring in-context learning and supervised fine-tuning
• Benchmarking LLMs on diverse process types and industries