Knowledge-Driven Hallucination in Large Language Models: An Empirical Study on Process Modeling

2025-09-18

AI Summary

This work addresses “knowledge-driven hallucination” in large language models (LLMs), wherein reliance on pre-trained knowledge causes them to disregard explicit input evidence during automated process modeling. Focusing on business process management (BPM), we design controlled experiments contrasting standard and atypical processes, deliberately constructing inputs where source evidence conflicts with model priors. We systematically evaluate LLMs’ faithfulness to source text when generating formal process models. We introduce the novel concept of knowledge-driven hallucination and propose a reliability assessment framework integrating qualitative analysis with quantitative fidelity metrics. Experimental results demonstrate that LLMs consistently override explicit input evidence to align outputs with internalized, standardized process patterns—revealing significant trustworthiness risks in high-stakes decision-support applications. Our methodology establishes a foundation for evidence-based verification of AI systems in process modeling.

Technology Category

• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The utility of Large Language Models (LLMs) in analytical tasks is rooted in their vast pre-trained knowledge, which allows them to interpret ambiguous inputs and infer missing information. However, this same capability introduces a critical risk of what we term knowledge-driven hallucination: a phenomenon where the model's output contradicts explicit source evidence because it is overridden by the model's generalized internal knowledge. This paper investigates this phenomenon by evaluating LLMs on the task of automated process modeling, where the goal is to generate a formal business process model from a given source artifact. The domain of Business Process Management (BPM) provides an ideal context for this study, as many core business processes follow standardized patterns, making it likely that LLMs possess strong pre-trained schemas for them. We conduct a controlled experiment designed to create scenarios with deliberate conflict between provided evidence and the LLM's background knowledge. We use inputs describing both standard and deliberately atypical process structures to measure the LLM's fidelity to the provided evidence. Our work provides a methodology for assessing this critical reliability issue and raises awareness of the need for rigorous validation of AI-generated artifacts in any evidence-based domain.

Problem

• Investigates knowledge-driven hallucination in LLMs
• Evaluates LLM reliability in automated process modeling
• Measures contradiction between source evidence and internal knowledge

Innovation

• Evaluating LLMs on automated process modeling
• Creating scenarios with evidence-knowledge conflicts
• Providing methodology for assessing reliability issues