ExOAR: Expert-Guided Object and Activity Recognition from Textual Data

2025-12-03

AI Summary

To address the challenge that unstructured text inadequately supports object-centric process mining, this paper proposes an interactive human-in-the-loop framework. First, a large language model (LLM) generates candidate object types, instances, and activities from textual input. Subsequently, domain-specific contextual information—such as users’ professional backgrounds—is integrated via staged prompting, expert feedback, and iterative refinement to ensure semantic accuracy. This approach overcomes the semantic ambiguity inherent in fully automated extraction, substantially improving both the accuracy and interpretability of structured event logs. Experiments conducted on activity-window data from five real users demonstrate that the resulting logs exhibit well-defined object–activity semantic associations, effectively bridging unstructured text with object-centric process analysis requirements. The framework establishes a novel paradigm for process mining in low-resource settings where high semantic fidelity is critical.

Technology Category

• Natural Language Processing: Information Extraction
• Data Mining & Knowledge Management: Intelligent Query Processing
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Bridging structured and unstructured data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Object-centric process mining requires structured data, but extracting it from unstructured text remains a challenge. We introduce ExOAR (Expert-Guided Object and Activity Recognition), an interactive method that combines large language models (LLMs) with human verification to identify objects and activities from textual data. ExOAR guides users through consecutive stages in which an LLM generates candidate object types, activities, and object instances based on contextual input, such as a user's profession, and textual data. Users review and refine these suggestions before proceeding to the next stage. Implemented as a practical tool, ExOAR is initially validated through a demonstration and then evaluated with real-world Active Window Tracking data from five users. Our results show that ExOAR can effectively bridge the gap between unstructured textual data and the structured log with clear semantics needed for object-centric process analysis, while it maintains flexibility and human oversight.

Problem

• Extracts objects and activities from unstructured text
• Combines LLMs with human verification for accuracy
• Bridges textual data to structured logs for process mining

Innovation

• LLM generates candidate objects and activities
• Human verification refines suggestions interactively
• Bridges unstructured text to structured semantic logs