Domain Adaptation of LLMs for Process Data

2025-09-03

AI Summary

This paper addresses the challenge of directly adapting process event logs to large language models (LLMs), which are inherently designed for natural language. To circumvent semantic distortion and excessive computational overhead associated with narrative-based log reconstruction, we propose a novel LLM-based approach that maps raw event log sequences directly into text-like token sequences—without natural language generation. Our method integrates parameter-efficient fine-tuning (e.g., LoRA) to preserve LLMs’ powerful sequential modeling capabilities while drastically reducing trainable parameters. It supports both single-task and multi-task predictive process monitoring. Empirically, our approach achieves superior prediction accuracy, faster convergence, and reduced hyperparameter tuning complexity. Extensive experiments on multiple benchmark datasets demonstrate consistent outperformance over state-of-the-art RNNs and narrative-generation-based LLM baselines—particularly in multi-task settings, where performance gains are most pronounced.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In recent years, Large Language Models (LLMs) have emerged as a prominent area of interest across various research domains, including Process Mining (PM). Current applications in PM have predominantly centered on prompt engineering strategies or the transformation of event logs into narrative-style datasets, thereby exploiting the semantic capabilities of LLMs to address diverse tasks. In contrast, this study investigates the direct adaptation of pretrained LLMs to process data without natural language reformulation, motivated by the fact that these models excel in generating sequences of tokens, similar to the objective in PM. More specifically, we focus on parameter-efficient fine-tuning techniques to mitigate the computational overhead typically associated with such models. Our experimental setup focuses on Predictive Process Monitoring (PPM), and considers both single- and multi-task predictions. The results demonstrate a potential improvement in predictive performance over state-of-the-art recurrent neural network (RNN) approaches and recent narrative-style-based solutions, particularly in the multi-task setting. Additionally, our fine-tuned models exhibit faster convergence and require significantly less hyperparameter optimization.

Problem

• Adapting pretrained LLMs to process data without natural language reformulation
• Using parameter-efficient fine-tuning to reduce computational overhead
• Improving predictive performance in multi-task process monitoring settings

Innovation

• Parameter-efficient fine-tuning for LLMs
• Direct adaptation without natural language reformulation
• Multi-task predictive process monitoring enhancement