Business as extit{Rule}sual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs

2025-05-24

AI Summary

Existing approaches inadequately model implicit business rule flows in commercial documents, particularly failing to capture logical dependencies (sequential, conditional, parallel) among rules. Method: We propose ExIde—a framework integrating prompt engineering, chain-of-thought reasoning, and dependency-aware fine-tuning—tailored for 12 state-of-the-art LLMs. Contribution/Results: ExIde introduces the first systematic modeling of inter-rule logical dependencies and establishes BPRF, the first Chinese business rule flow annotation dataset (50 documents, 326 rules). It designs a joint rule-pair representation and dependency labeling paradigm and builds the first LLM evaluation benchmark specifically for rule flow understanding. Experiments on BPRF show ExIde improves rule extraction F1 by 18.7% and achieves up to 89.3% accuracy in dependency classification, significantly demonstrating the effectiveness and interpretability of LLMs in modeling business rule flows.

Technology Category

• Natural Language Processing: Information Extraction
• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Data Mining & Knowledge Management: Rule Mining & Pattern Mining

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Process mining aims to discover, monitor and optimize the actual behaviors of real processes. While prior work has mainly focused on extracting procedural action flows from instructional texts, rule flows embedded in business documents remain underexplored. To this end, we introduce a novel annotated Chinese dataset, extbf{BPRF}, which contains 50 business process documents with 326 explicitly labeled business rules across multiple domains. Each rule is represented as apair, and we annotate logical dependencies between rules (sequential, conditional, or parallel). We also propose extbf{ExIde}, a framework for automatic business rule extraction and dependency relationship identification using large language models (LLMs). We evaluate ExIde using 12 state-of-the-art (SOTA) LLMs on the BPRF dataset, benchmarking performance on both rule extraction and dependency classification tasks of current LLMs. Our results demonstrate the effectiveness of ExIde in extracting structured business rules and analyzing their interdependencies for current SOTA LLMs, paving the way for more automated and interpretable business process automation.

Problem

• Extracting business rule flows from documents using LLMs
• Identifying logical dependencies between business rules
• Benchmarking LLM performance on rule extraction tasks

Innovation

• Annotated Chinese dataset BPRF for business rules
• ExIde framework for rule extraction using LLMs
• Evaluates 12 SOTA LLMs on BPRF dataset